Lead Data Engineer, Databricks

New
J
JobgetherData engineering
Based in the United StatesFull-TimeLead
Salary not disclosed
Apply NowOpens the employer's application page

Job Details

Experience
At least 7 years of professional data engineering experience, including experience leading projects or engineering teams.
Required Skills
AWSPythonSQLGCPGitMicrosoft AzureSparkDatabricksPySpark

Requirements

  • Have at least 7 years of professional data engineering experience, including experience leading projects or engineering teams.
  • Demonstrate strong proficiency in Python, SQL, Apache Spark, and Delta Lake.
  • Have hands-on production experience building batch and streaming data pipelines at enterprise scale.
  • Have practical experience with at least one major cloud platform: AWS, Microsoft Azure, or Google Cloud Platform, including its data services.
  • Have experience with data modeling, Git, and CI/CD practices for data engineering workloads.
  • Be able to optimize Spark performance, improve pipeline efficiency, and manage platform costs.
  • Understand scalable data architecture and modern data engineering practices.
  • Be able to explain complex technical concepts and trade-offs to non-technical stakeholders.
  • Be able to guide engineers, provide constructive feedback, and own technical delivery.
  • Nice to have experience in healthcare or life sciences environments.
  • Nice to have a Databricks Certified Data Engineer Associate or Professional certification.
  • Nice to have experience with Unity Catalog, Lakeflow, or Databricks Asset Bundles.
  • Nice to have familiarity with dbt, Airflow, or equivalent orchestration and data transformation tools.
  • Nice to have experience migrating legacy ETL systems to modern cloud-based data platforms or previous consulting or client-facing engineering experience.

Responsibilities

  • Lead data engineering activities across customer engagements from initial design and implementation through production handover.
  • Build, maintain, and optimize batch and streaming pipelines using PySpark, SQL, and Delta Lake on Databricks.
  • Implement Medallion architecture across cloud environments and establish consistent data layer conventions and engineering patterns.
  • Benchmark Spark jobs, tune SQL queries, right-size compute resources, and evaluate technical trade-offs to improve performance and cost efficiency.
  • Implement data quality and validation processes as code, including Lakeflow expectations and automated testing within CI pipelines.
  • Establish and maintain engineering standards for the reliability, scalability, and quality of delivered solutions.
  • Mentor junior and mid-level engineers through code reviews, collaborative development, technical guidance, and design feedback.
  • Partner with Data Architects to validate architectural decisions and identify potential design limitations during implementation.
  • Communicate technical decisions, delivery progress, and implementation trade-offs to customers and non-technical stakeholders.
  • Work with customer teams and platform specialists to deliver enterprise data solutions.
View Full Description & ApplyYou'll be redirected to the employer's site
View details
Apply Now