- Design and implement batch processing pipelines using Spark (Python or Scala) and SQL.
- Design and implement streaming ETL/ELT processes from a diverse set of data sources.
- Write code supporting the development of big data solutions, implementing complete data integration or analytic use cases.
- Design and implement APIs in any modern Python API framework.
- Communicate effectively with our Business Analysis teams to ensure alignment with business requirements.
- Write representative end-to-end and functional tests using open-source tools.
- Implement monitoring solutions for our data platform through alerts and dashboards.
PythonSQLETL+2 more