A
Senior Data Engineer
On-siteFull-timeData EngineeringCloud ServicesIT Consulting
Join a cloud services team to design, develop, and maintain an AWS- and Databricks-based data platform integrating Snowflake, Delta Lake, and analytics engineering.
Responsibilities
- Develop and maintain Python/PySpark ETL and ELT pipelines on Databricks and Snowflake.
- Implement Bronze, Silver, and Gold data layers using Medallion architecture and Delta Lake.
- Manage Databricks SQL Warehouses, query optimization, and data exploration.
- Use AWS services for data storage, compute, and secure data movement.
- Perform database administration and performance tuning across Delta Lake and relational or warehouse environments.
- Design Gold-layer data models that support Power BI, Tableau, and other BI tools.
- Manage metadata, permissions, lineage, and access controls with Unity Catalog.
- Implement automated data quality tests within CI/CD data pipelines.
- Troubleshoot distributed data processes and database bottlenecks while collaborating with analysts, product managers, and data scientists.
Requirements
- At least 5 years of experience as a Data Engineer.
- Hands-on experience with AWS services such as S3, IAM, and EC2, plus Snowflake or another modern cloud data warehouse.
- At least 1 year of intensive hands-on Databricks experience, including Notebooks and Workflows.
- Strong PySpark or Spark Scala skills and advanced SQL, including window functions and CTEs.
- Experience with Delta Lake, Parquet, cloud storage, and AWS S3 or Azure ADLS.
- Experience designing fact and dimension tables, including star or Snowflake schemas.
- Experience with database performance tuning, query profiling, indexing, and partition pruning.
Nice to have
- Experience with Delta Live Tables or dbt on Databricks or Snowflake.
- Familiarity with MLOps concepts and MLflow.
- AWS, Snowflake, or Databricks certifications.
- Experience with Apache Airflow or other orchestration tools.