Lead Data Engineer, Databricks

V
V4CData engineering
United StatesFull-TimeLead
Salary not disclosed
Apply NowOpens the employer's application page

Job Details

Experience
7+ years of data engineering plus experience leading projects or teams
Required Skills
AWSPythonSQLAzureSparkCI/CDDatabricksPySpark

Requirements

  • Have 7+ years of data engineering experience and experience leading projects or teams.
  • Demonstrate strong proficiency in Python, SQL, Apache Spark, and Delta Lake.
  • Have production experience building batch and streaming pipelines at enterprise scale.
  • Have hands-on experience with AWS, Azure, or GCP and its data services.
  • Have experience with data modeling, Git, and CI/CD for data workloads.
  • Demonstrate experience optimizing Spark performance and managing platform cost.
  • Communicate technical trade-offs to non-technical stakeholders.
  • Healthcare or life sciences experience is preferred but not required.
  • Databricks Certified Data Engineer Associate or Professional certification is preferred.
  • Experience with Unity Catalog, Lakeflow, or Databricks Asset Bundles is preferred.
  • Experience with dbt, Airflow, or equivalent orchestration and transformation tooling is preferred.
  • Experience migrating a legacy ETL estate to a modern cloud platform, or a consulting or client-facing engineering background, is preferred.

Responsibilities

  • Lead engineering on customer engagements from design through production handover.
  • Build and optimize batch and streaming pipelines using PySpark, SQL, and Delta Lake on Databricks.
  • Implement Medallion architecture across cloud ecosystems and establish layer conventions for the team.
  • Benchmark Spark jobs, tune SQL, right-size compute, and explain performance and cost trade-offs to customers.
  • Implement data quality and validation as code, including Lakeflow expectations and automated testing in CI.
  • Mentor mid-level and junior engineers through code review, pairing, and design feedback.
  • Partner with the Data Architect on design decisions and escalate when implementation challenges the design.
View Full Description & ApplyYou'll be redirected to the employer's site
View details
Apply Now