L
Senior Data Platform Engineer
היברידימשרה מלאהData Engineering
Join a data platform team in Tel Aviv, working in a hybrid model with three office days per week. Own and evolve the infrastructure supporting large-scale batch and streaming data workloads, with a strong focus on reliability, governance, performance, cost, automation, and AI-enabled workflows.
תחומי אחריות
- Administer and evolve Databricks, including Unity Catalog, compute, governance, jobs, and cost controls.
- Own Confluent Kafka, CDC pipelines, Airflow, Elasticsearch, databases, S3 data assets, and the underlying Kubernetes cluster.
- Design, build, and operate large-scale batch and streaming data pipelines in production.
- Improve data-platform reliability, performance, governance, observability, testing, documentation, and cost management.
- Drive architecture decisions across Databricks, Kafka, Elasticsearch, and AWS.
- Automate manual platform processes and develop safe self-service tooling for data teams.
- Build and integrate LLM and agent tooling, including MCP servers and agent interfaces, into platform workflows.
- Lead technical initiatives across teams from initial definition through implementation and adoption.
- Provide operational ownership for shipped systems, including troubleshooting and incident resolution.
דרישות
- At least 5 years of hands-on experience building and operating large-scale production data pipelines, including batch and streaming workloads.
- Deep experience with Spark and Databricks, including Delta Lake, Unity Catalog, job and cluster tuning, and Structured Streaming or Delta Live Tables.
- Experience with Kafka-based streaming and CDC, including Confluent Kafka, Debezium or equivalent tools, schema evolution, and failure modes.
- Experience writing Airflow DAGs and operating Airflow platforms.
- Expertise in Python and SQL, with a focus on clean, tested, and performant code.
- Strong data modeling and architecture judgment, including the ability to explain scalability, performance, and cost tradeoffs.
- Working knowledge of AWS, including S3, IAM, and networking fundamentals, plus Terraform, Docker, Kubernetes, and CI/CD.
- Experience leading cross-team technical initiatives from ambiguous requirements through delivery and adoption.
- Daily hands-on use of AI coding tools and experience integrating LLM and agent tooling, including MCP, embeddings, or vector search, into real workflows.
יתרון
- Experience operating Elasticsearch at scale, including indexing pipelines, cluster operations, reindexing, and rollout strategies.
- Experience with FinOps or cloud cost management for data platforms.
- Experience with Vault or comparable secrets-management tools and advanced Kubernetes operations.
- Experience improving inherited systems with significant technical debt.