J
Senior Data Engineer
Not specifiedFull-timeData Engineering
Build and operate production data and machine learning pipelines for a partner company in Israel. The role focuses on Python, AWS, ETL, workflow orchestration, and scalable ML infrastructure, in collaboration with Data Scientists and ML Engineers.
Responsibilities
- Build, maintain, and improve production data and ML pipelines for forecasting and data-driven decision-making.
- Develop scalable, reliable, and maintainable Python ETL and data-processing workflows.
- Design and orchestrate workflows using Apache Airflow and AWS Step Functions.
- Use AWS Glue for data processing and transformation.
- Support ML model deployment and operations using AWS SageMaker.
- Collaborate with Data Scientists and ML Engineers to put models and analytical solutions into production.
- Monitor pipeline health, troubleshoot production issues, and improve reliability, scalability, and efficiency.
- Contribute to architecture and technical decisions for the data platform and ML infrastructure.
- Work with distributed cross-functional Agile teams to improve engineering practices and delivery.
Requirements
- Strong professional experience using Python for data engineering and production systems.
- Experience building and maintaining production data or machine learning pipelines.
- Hands-on experience with Apache Airflow, AWS Step Functions, and AWS Glue.
- Experience deploying and supporting ML models with AWS SageMaker.
- Strong hands-on experience with AWS and cloud-based production environments.
- Experience with large-scale systems and reliability and performance considerations.
- Excellent written and verbal English communication skills.
Nice to have
- Experience with GCP.
- Familiarity with monitoring and observability tools.
Benefits
- Shared responsibility and decision-making within a collegial team.
- Agile environment with opportunities to influence technical direction.
- Training, mentoring, and professional development.
- Opportunities to work across projects and grow professionally.
- Possibility of business travel.
- Flexible arrangements suited to a distributed team.