- Design, build, maintain and primarily operate scalable streaming and batch data pipelines, with a strong focus on maintenance, monitoring, troubleshooting and continuous improvement of existing pipelines.
- Work with AWS services, including Redshift, EMR and ECS, to support data processing and analytics workloads.
- Develop and maintain data workflows using Python and SQL.
- Orchestrate and monitor pipelines using Apache Airflow.
- Build and deploy containerized applications using Docker and Kubernetes.
- Break down high-level system designs into well-defined, deliverable tasks with realistic estimates.
- Collaborate with cross-functional teams in a fast-paced and distributed environment across the US and Europe.
- Drive automation, observability and monitoring to improve reliability, performance and operational efficiency.
- Support knowledge transfer and ownership handover as part of the planned transition to the consuming team.
AWSDockerPython+5 more