Senior Software Engineer, Data & Platform Services
New
Globally distributed Data Platform team in Europe, Must be able to collaborate with teams in the US, Europe, Israel, and India.Full-TimeSenior
Salary not disclosed
Apply NowOpens the employer's application page
Job Details
- Experience
- 6–8 years
- Required Skills
- PythonSQLApache AirflowETLGCPHadoopKubernetesSparkDatabricks
Requirements
- 6–8 years of experience building and supporting distributed systems and large-scale data platforms.
- Proven experience migrating enterprise data platforms from legacy technologies like Hadoop, Hive, or Impala to Databricks.
- Strong knowledge of the Databricks ecosystem, including Jobs, Delta Lake, and Spark SQL.
- Advanced Python programming skills for ETL development and orchestration.
- Extensive experience designing and optimizing data pipelines using Airflow.
- Strong proficiency with Google Cloud Platform (GCP), GCS, and Kubernetes (GKE) containerized architectures.
- Understanding of distributed data processing architectures and batch versus real-time trade-offs.
- Experience using AI-powered development tools like Claude or Codex for code modernization.
- Strong collaboration skills for working in globally distributed teams across multiple time zones.
Responsibilities
- Lead the modernization of the data platform by migrating mission-critical pipelines from Hadoop, Spark, and Impala to a cloud-native Databricks architecture.
- Architect ETL workflows, convert SQL workloads, and manage complex integrations and secrets.
- Leverage AI coding assistants to accelerate code modernization, validation, and quality control.
- Design and optimize scalable, resilient data pipelines using Airflow and Databricks.
- Implement comprehensive data validation and testing strategies to maintain data integrity.
- Enhance platform reliability through improved observability, monitoring, incident response, and defined SLOs/SLAs.
View Full Description & ApplyYou'll be redirected to the employer's site
