J
Senior Data Infrastructure Engineer
RemoteFull-timeData Engineering
A partner company is seeking a senior engineer to design and scale web scraping and data ingestion systems that process millions of records daily. The role is fully remote.
Responsibilities
- Design and manage web scraping infrastructure, including proxy networks, TLS fingerprinting, browser automation, and session rotation
- Build fault-tolerant, high-throughput scraping pipelines for social media and other web sources
- Develop distributed queues and workflow orchestration systems
- Architect storage for structured and unstructured data
- Implement monitoring, alerts, and recovery mechanisms for platform changes, blocking patterns, and proxy failures
- Maintain reliable, high-volume data ingestion and enable downstream AI and analytics workflows
Requirements
- At least 4 years of experience in high-volume web scraping, data engineering, or reverse engineering
- Strong experience with anti-bot systems, TLS impersonation, browser automation, and proxy management
- Advanced proficiency in Python or Go
- Experience with browser automation and scraping frameworks such as Playwright, Puppeteer, Scrapy, or Selenium
- Practical knowledge of distributed systems and task queues, including Temporal, Ray, Kafka, and Redis
- Strong SQL skills and experience with analytical data lakes and data warehouses
- Ability to design scalable, fault-tolerant systems and troubleshoot data ingestion challenges
Nice to have
- Experience extracting short-form video content and user metadata from TikTok, Instagram, and YouTube
- Experience integrating scraping outputs with vector databases and real-time AI processing queues
Benefits
- Fully remote work arrangement