J
Senior Data Engineer - Web Scraping
·Israel
RemoteFull-timeData EngineeringMarketplace Platforms
A remote senior data engineering role focused on designing and maintaining web scrapers and data pipelines. You will use Python, Pandas, and Airflow to process large datasets and ensure reliable data delivery. The position offers autonomy, collaboration, and the opportunity to work on sophisticated data products.
Responsibilities
- Design, develop, deploy, and maintain web scrapers to collect alternative datasets.
- Use Python and Pandas to clean, explore, transform, and manipulate large datasets.
- Build and maintain efficient data pipelines that ingest scraped data into databases.
- Develop and manage scheduled workflows using Apache Airflow.
- Collaborate with analysts to understand data requirements and translate them into solutions.
- Develop quality-control checks to validate data availability and integrity.
- Maintain alerting systems and resolve data incidents.
- Design tools and automation to improve web-scraping platform efficiency.
Requirements
- Bachelor's or master's degree in Computer Science or related field.
- 4-6 years of professional experience in data engineering.
- Strong programming skills in Python and SQL.
- Advanced expertise with Python Pandas library.
- Strong web scraping experience with Selenium, Scrapy, and related tools.
- Strong experience with Apache Airflow.
- Understanding of web technologies including HTML, JavaScript, and APIs.
- Proven experience working with large datasets.
- Strong communication and collaboration skills.
- Self-motivated and detail-oriented with ability to work independently.
Nice to have
- Experience with Docker and Kubernetes.
- Familiarity with CI/CD technologies such as Jenkins and GitHub Actions.
- Experience with AWS services like S3, RDS, SNS, SQS, and Lambda.
Benefits
- Fully remote position with flexibility.
- Full-time opportunity in a collaborative environment.
- Significant autonomy and ownership.
- Exposure to complex datasets and investment-related decisions.
- Collaboration with engineering, product, and data professionals.
- Innovative and continuously improving environment.