Lead Data Engineer, Databricks
New
J
JobgetherData engineering
Based in the United StatesFull-TimeLead
Salary not disclosed
Apply NowOpens the employer's application page
Job Details
- Experience
- At least 7 years of professional data engineering experience, including experience leading projects or engineering teams.
- Required Skills
- AWSPythonSQLGCPGitMicrosoft AzureSparkDatabricksPySpark
Requirements
- Have at least 7 years of professional data engineering experience, including experience leading projects or engineering teams.
- Demonstrate strong proficiency in Python, SQL, Apache Spark, and Delta Lake.
- Have hands-on production experience building batch and streaming data pipelines at enterprise scale.
- Have practical experience with at least one major cloud platform: AWS, Microsoft Azure, or Google Cloud Platform, including its data services.
- Have experience with data modeling, Git, and CI/CD practices for data engineering workloads.
- Be able to optimize Spark performance, improve pipeline efficiency, and manage platform costs.
- Understand scalable data architecture and modern data engineering practices.
- Be able to explain complex technical concepts and trade-offs to non-technical stakeholders.
- Be able to guide engineers, provide constructive feedback, and own technical delivery.
- Nice to have experience in healthcare or life sciences environments.
- Nice to have a Databricks Certified Data Engineer Associate or Professional certification.
- Nice to have experience with Unity Catalog, Lakeflow, or Databricks Asset Bundles.
- Nice to have familiarity with dbt, Airflow, or equivalent orchestration and data transformation tools.
- Nice to have experience migrating legacy ETL systems to modern cloud-based data platforms or previous consulting or client-facing engineering experience.
Responsibilities
- Lead data engineering activities across customer engagements from initial design and implementation through production handover.
- Build, maintain, and optimize batch and streaming pipelines using PySpark, SQL, and Delta Lake on Databricks.
- Implement Medallion architecture across cloud environments and establish consistent data layer conventions and engineering patterns.
- Benchmark Spark jobs, tune SQL queries, right-size compute resources, and evaluate technical trade-offs to improve performance and cost efficiency.
- Implement data quality and validation processes as code, including Lakeflow expectations and automated testing within CI pipelines.
- Establish and maintain engineering standards for the reliability, scalability, and quality of delivered solutions.
- Mentor junior and mid-level engineers through code reviews, collaborative development, technical guidance, and design feedback.
- Partner with Data Architects to validate architectural decisions and identify potential design limitations during implementation.
- Communicate technical decisions, delivery progress, and implementation trade-offs to customers and non-technical stakeholders.
- Work with customer teams and platform specialists to deliver enterprise data solutions.
View Full Description & ApplyYou'll be redirected to the employer's site