F
Senior Data Engineer – Databricks & AI/ML
·Israel
On-siteFull-timeData EngineeringFinTech
Own the full data lifecycle, from pipeline architecture to feature engineering, model-ready datasets, and inference infrastructure. Build robust ETL/ELT pipelines on Databricks and AWS, and integrate LLM-based automation. Collaborate with engineering and product to turn data into production intelligence.
Responsibilities
- Design, build, and maintain scalable data pipelines and ETL/ELT workflows on Databricks (Delta Lake, Spark, Workflows/Jobs)
- Architect and manage data lakehouse infrastructure across Databricks and AWS (S3, Redshift, Glue, etc.)
- Build feature pipelines and model-ready datasets to support machine learning and predictive modeling
- Develop and deploy predictive ML models and support experimentation pipelines
- Build and integrate RAG pipelines, vector retrieval, and LLM-based automation into internal tools and data workflows
- Integrate AI/ML capabilities with internal APIs, services, and data platforms
- Implement data quality, observability, and monitoring practices across pipelines and AI workloads
- Optimize pipeline performance, cost, and reliability at scale
- Collaborate with engineering and product teams to identify where better data infrastructure and AI can improve internal tools and decision systems
- Contribute to architectural decisions around the company's data and AI platform strategy
Requirements
- 4+ years of experience in data engineering, software engineering, or a related field, with hands-on Databricks experience (Spark, Delta Lake, Unity Catalog, Workflows)
- Strong programming skills in Python and SQL
- Experience designing and operating production-grade data pipelines on large datasets
- Experience working with AWS (S3, Redshift, Glue, Lambda, or similar)
- Solid understanding of ML fundamentals (feature engineering, model training/evaluation) as applied to data pipeline design
- Familiarity with LLM APIs and modern AI frameworks, and how to integrate them into data-driven systems
- Experience with data modeling, warehousing, and lakehouse architecture
- Understanding of CI/CD, orchestration, and monitoring for data and ML workloads
- Strong system design and problem-solving skills
- Ability to take ownership end-to-end and collaborate across data, ML, and product teams
Nice to have
- Experience building LLM-based agents, RAG architectures, or AI automation systems
- Familiarity with LangChain, LangGraph, or similar frameworks
- Experience with vector databases and retrieval systems
- Experience with distributed systems or large-scale batch/streaming data processing (Spark Structured Streaming, Kafka)
- Databricks certifications (Data Engineer Associate/Professional, ML) a plus
- Experience with dbt, Airflow, or similar orchestration tools
Benefits
- Own the data foundation that powers production AI systems used across the organization
- Work with modern data platforms (Databricks, AWS) and cutting-edge AI/LLM technologies
- Join a highly technical, full-stack engineering team
- Help shape the company's data and AI infrastructure from the ground up