N
Senior Data Engineer
לא צויןמשרה מלאהData EngineeringAI InfrastructureData InfrastructureEnterprise Software
Lead the design, implementation, and operation of reliable production data pipelines for analytics and machine learning. This hands-on role uses Python, SQL, workflow orchestration, cloud infrastructure, Docker, and Kubernetes, with a focus on resilient and cost-efficient data processing.
תחומי אחריות
- Design, build, and own production-grade data pipelines using Python and SQL
- Develop resilient, stateless, and idempotent pipelines that handle retries, failures, and infrastructure interruptions
- Implement data transformations, validation, and data quality checks
- Optimize pipelines for performance, reliability, and cost efficiency
- Collaborate with analytics, data science, and machine learning teams to deliver trusted datasets
- Orchestrate pipelines with Airflow or an equivalent workflow framework
- Package and run workloads with Docker on Kubernetes
- Build CI/CD automation for data pipelines
- Use infrastructure as code to manage pipeline runtime infrastructure
דרישות
- At least 8 years of experience as a data engineer, primarily focused on data pipelines
- At least 6 years of hands-on Python and SQL experience
- At least 3 years of experience running workloads on Kubernetes
- Strong understanding of stateless system design and idempotent data processing
- Experience building and operating data pipelines in cloud environments
- Experience with workflow orchestration frameworks
- Strong Linux fundamentals and production debugging skills
- Working knowledge of spoken and written English
יתרון
- Open-source software experience
- Apache Spark or similar distributed processing experience
- Experience supporting machine learning workflows
- Familiarity with Spot or preemptible compute
- Experience with relational and non-relational data stores
- Experience with large-scale or high-reliability data systems
- Experience collaborating with data science and machine learning teams
הטבות
- Competitive compensation
- Career growth and learning opportunities
- Flexibility and ownership
- Collaborative and innovative culture
- Opportunity to work on impactful AI projects
- International work environment