- Architect, build, and maintain the core infrastructure behind our large-scale asynchronous data collection platform.
- Design, implement, and continuously improve sophisticated anti-blocking strategies, including browser fingerprinting, proxy rotation, CAPTCHA handling, and other techniques.
- Design, develop, test, and maintain robust scraping components using Python and modern scraping frameworks such as Playwright, Scrapy, Selenium, Requests, and related tools.
- Build monitoring, alerting, and logging systems that help identify issues quickly and continuously improve scraper reliability and data quality.
- Develop and maintain scalable data ingestion pipelines and integrations with internal and external REST APIs.
- Contribute to infrastructure automation using Docker, CI/CD pipelines, Linux environments, and related DevOps practices.
- Work closely with other engineers to improve our scraping platform, establish engineering standards, and mentor less experienced teammates.
DockerPythonSQL+5 more