J
Lead Data Engineer, Data Crawling and Ingestion
Not specifiedFull-timeData EngineeringEnterprise SoftwareCybersecurity
Lead a team of five data engineers responsible for large-scale data pipelines and ingestion systems. Set technical direction, make architectural decisions, and contribute hands-on in a complex, data-intensive software environment.
Responsibilities
- Lead, mentor, and support a team of five data engineers, including task prioritization, technical guidance, and career development.
- Own the technical direction and architecture of large-scale data pipeline systems.
- Design and oversee solutions for complex and continuously changing internet data sources.
- Ensure data systems are scalable, resilient, and efficient at high volumes.
- Collaborate with DevOps, infrastructure, security, and product teams on technical solutions.
- Drive best practices for code quality, observability, documentation, and operational excellence.
- Participate in hands-on development and technical reviews.
Requirements
- Experience leading or mentoring engineers on a technical team.
- Strong experience with relational and/or NoSQL databases, including schema design and performance optimization.
- Solid understanding of distributed systems and data-intensive applications.
- Experience designing or maintaining batch and/or streaming data pipelines.
- Familiarity with change data capture, ETL mechanisms, and large-scale multi-job orchestration systems.
- Experience with Python.
- Strong problem-solving skills in complex and evolving environments.
Nice to have
- Experience with software package ecosystems, package registries, dependency metadata, and versioning concepts.
- Experience working with internet-scale data sources or ingestion platforms.