Data Collection Engineer
New
T
Two Six TechnologiesData Engineering
Remote, USAFull-TimeSenior
Salary160,000 - 200,000 USD per year
Apply NowOpens the employer's application page
Job Details
- Experience
- 7+ years
- Required Skills
- PythonElasticSearchKubernetesDistributed Systems
Requirements
- 7+ years of professional software engineering experience.
- Expert-level proficiency in Python.
- Deep experience with Scrapy and distributed scraping architectures.
- Proven experience with browser automation tools (Playwright, Selenium, or Puppeteer).
- Mastery of JSON, JSON Schema, and Pydantic.
- Hands-on experience with Elasticsearch clusters.
- Strong proficiency in managing and scaling applications within Kubernetes environments.
- Experience building structured pipeline workflows for complex data extraction.
- Strong reverse-engineering skills, including analyzing network traffic and uncovering hidden APIs.
- Bachelor’s degree in Computer Science or Engineering.
Responsibilities
- Design and deploy high-performance, distributed web scrapers using Python and Scrapy.
- Utilize browser scripting tools to extract data from modern, dynamic, and JavaScript-heavy websites.
- Deploy, scale, and manage scraping workloads on Kubernetes.
- Define strict JSON schemas and use Pydantic to enforce data types and validate payloads.
- Build and optimize search and storage pipelines using Elasticsearch.
- Architect robust pipeline workflows to manage the end-to-end data lifecycle.
- Manage complex proxy rotation, session handling, and browser fingerprinting.
View Full Description & ApplyYou'll be redirected to the employer's site
