Software Engineer II (Data Harvesting)
New
India Remote, 1:00 PM – 9:00 PM IST (Monday–Friday)Full-TimeMiddle
Salary not disclosed
Apply NowOpens the employer's application page
Job Details
- Experience
- 3–5 years of professional experience
- Required Skills
- AWSBackend DevelopmentDockerPostgreSQLPythonSQLETLData StructuresSeleniumSparkRESTful APIsPlaywright
Requirements
- 3–5 years of professional experience in backend or data engineering
- Strong proficiency in Python
- Experience working with large-scale data ingestion systems
- Experience building and maintaining data pipelines or backend services
- Familiarity with web data extraction concepts, such as: APIs, web scraping (Selenium, Playwright, or similar) and handling structured and unstructured data
- Strong SQL skills (PostgreSQL or similar databases)
- Experience with Apache Spark or similar data processing frameworks
- Experience working in a AWS cloud environment
- Familiarity with Docker or containerization
Responsibilities
- Contribute to building systems and frameworks that capture web data at scale, including working with structured and unstructured data sources
- Design and develop data extraction components using tools such as APIs, scraping frameworks, and parsing logic
- Build and maintain ETL/ELT pipelines using technologies like Apache Spark and cloud platforms (preferably AWS)
- Write clean, efficient Python code to support data ingestion, transformation, and processing workflows
- Help improve the reliability and performance of data pipelines through monitoring, debugging, and optimization
- Work with senior engineers to enhance systems that handle: data quality and normalization, large-scale data ingestion and pipeline scalability
- Troubleshoot issues related to: data inconsistencies, pipeline failures and source data changes (e.g., website structure updates)
- Collaborate with product, data, and engineering teams to ensure data is usable and aligned with business needs
- Contribute to documentation and participate in code reviews to support engineering best practices
View Full Description & ApplyYou'll be redirected to the employer's site
