Senior Software Engineer, Data & Platform Services
New
L
LivePersonConversational AI
Fully Remote (#LI-Remote), Globally distributed environment with colleagues across the US, Europe, Israel, and India.Full-TimeSenior
Salary not disclosed
Apply NowOpens the employer's application page
Job Details
- Experience
- 6–8 years
- Required Skills
- PythonApache AirflowETLGCPHadoopKubernetesDatabricks
Requirements
- 6–8 years of experience building and supporting distributed systems and large-scale data platforms.
- Proven experience migrating enterprise data platforms from legacy technologies like Hadoop, Hive, or Impala to Databricks.
- Deep expertise in the Databricks ecosystem, including Jobs, Delta Lake, and Spark SQL.
- Advanced Python programming skills for ETL development.
- Experience designing, building, and optimizing production data pipelines using Airflow.
- Experience with Google Cloud Platform (GCP) and Google Cloud Storage (GCS).
- Understanding of containerized environments and Kubernetes-based architectures (GKE).
- Experience using AI-powered development tools for code modernization and migration.
- Strong communication skills for working within globally distributed teams.
Responsibilities
- Lead the migration of mission-critical data pipelines from Hadoop, Spark, and Impala to a cloud-native Databricks ecosystem.
- Leverage AI coding assistants like Claude and Codex to modernize ETL scripts and SQL workloads.
- Design and optimize scalable data workflows using Airflow and Databricks.
- Implement comprehensive testing and validation strategies to ensure data quality and integrity.
- Enhance platform observability, monitoring, and incident response practices through SLOs and on-call rotations.
View Full Description & ApplyYou'll be redirected to the employer's site
