Research Crawling Engineer
New
T
True Talent Solutions PartnersArtificial Intelligence
This role is fully remote. The primary anchor location is Los Angeles, CA, United States, though candidates in other US cities are also welcome.Full-TimeSenior
Salary160,000 - 250,000 USD per year
Apply NowOpens the employer's application page
Job Details
- Experience
- 4+ years (2+ years for PhD holders)
- Required Skills
- PythonJavaC++GoRustPlaywright
Requirements
- 4+ years of experience building and operating web crawlers at scale (2+ years for candidates with a PhD).
- At least 3 years hands-on with production-grade crawling systems.
- Proficiency in one or more of: Go, Rust, Python, Java, or C++.
- Experience operating data pipelines at TB+ scale with distributed or parallel processing systems.
- Deep knowledge of HTTP, networking, and browser behavior.
- Proven experience bypassing anti-bot systems, handling rate limits, and working with JS-heavy sites.
- Familiarity with proxy systems, IP rotation, or request orchestration.
- Experience with headless browsers such as Playwright, Puppeteer, or Chrome DevTools Protocol.
- Background in NLP pipelines, ML dataset curation, or work at an AI lab is a bonus.
- Experience with data quality evaluation, scoring, or benchmarking at scale is a bonus.
Responsibilities
- Build and maintain large-scale web crawlers across diverse domains including social media, travel, and multi-language sites to power AI training datasets.
- Design high-throughput, fault-tolerant data collection systems capable of handling millions to billions of URLs per day.
- Tackle anti-bot systems, rate limits, and dynamic or JS-heavy sites, finding creative solutions when standard protocols fall short.
- Develop pipelines for cleaning, deduplication, filtering, and normalization of web data at TB to PB scale.
- Collaborate with research teams to align data collection strategies with modeling and dataset curation needs.
- Monitor crawl performance, coverage, and data quality, iterating rapidly as web environments evolve.
- Optimize cloud and bare-metal infrastructure for cost, latency, and reliability.
View Full Description & ApplyYou'll be redirected to the employer's site
