Senior Data Infrastructure Engineer (Scraping & Scale)
New
J
JobgetherData infrastructure
Based in United StatesFull-TimeSenior
Salary not disclosed
Apply NowOpens the employer's application page
Job Details
- Experience
- At least 4 years of experience in high-volume web scraping, data engineering, or reverse engineering.
- Required Skills
- PythonSQLApache KafkaGoDistributed Systems
Requirements
- Have at least 4 years of experience in high-volume web scraping, data engineering, or reverse engineering.
- Bring strong experience handling advanced anti-bot systems, including Cloudflare, Akamai, and PerimeterX.
- Use TLS impersonation, browser automation, and proxy management.
- Have advanced proficiency in Python or Go.
- Have extensive experience with browser automation and scraping frameworks such as Playwright, Puppeteer, Scrapy, or Selenium.
- Have practical knowledge of distributed systems and task queues, including Temporal, Ray, Kafka, and Redis.
- Have strong SQL skills and experience with modern analytical data lakes and data warehouses.
- Be able to design scalable, fault-tolerant systems and troubleshoot complex data ingestion challenges.
- Preferred: Experience extracting short-form video content and user metadata from TikTok, Instagram, and YouTube.
- Preferred: Experience integrating scraping outputs with vector databases and real-time AI processing queues.
Responsibilities
- Design and manage web scraping infrastructure using residential proxy networks, TLS fingerprinting, browser automation, and session rotation.
- Build fault-tolerant, high-throughput scraping pipelines for Instagram, TikTok, YouTube, X, and other web sources.
- Develop distributed queues and workflow orchestration systems using Temporal, Ray, Apache Kafka, and Celery.
- Architect structured and unstructured data storage using Parquet, Apache Iceberg, Snowflake, and Amazon S3.
- Implement monitoring, automated alerts, and recovery mechanisms for platform changes, blocking patterns, and proxy failures.
- Maintain reliable data ingestion at scale while minimizing downtime and processing millions of records daily.
- Enable downstream AI and analytics workflows through robust data infrastructure.
View Full Description & ApplyYou'll be redirected to the employer's site
