Lead Data Engineer, Databricks

New
V
V4CData engineering consulting
Remote, US; Source API remote eligibility restrictions: United StatesFull-TimeLead
Salary not disclosed
Apply NowOpens the employer's application page

Job Details

Experience
7+ years of data engineering plus experience leading projects or teams
Required Skills
AWSPythonSQLGCPAzureSparkCI/CDDatabricksPySpark

Requirements

  • Have 7+ years of data engineering experience and experience leading projects or teams.
  • Demonstrate strong proficiency in Python, SQL, Apache Spark, and Delta Lake.
  • Have production experience building both batch and streaming pipelines at enterprise scale.
  • Have hands-on experience with AWS, Azure, or GCP and its data services.
  • Have experience with data modeling, Git, and CI/CD for data workloads.
  • Demonstrate experience optimizing Spark performance and managing platform cost.
  • Have strong client communication skills, including the ability to explain technical trade-offs to non-technical stakeholders.
  • Healthcare or life sciences experience is preferred but not required.
  • Databricks Certified Data Engineer Associate or Professional certification is preferred.
  • Hands-on experience with Unity Catalog, Lakeflow, or Databricks Asset Bundles is preferred.
  • Experience with dbt, Airflow, or equivalent orchestration and transformation tooling is preferred.
  • Experience migrating a legacy ETL estate to a modern cloud platform is preferred.
  • Consulting or client-facing engineering experience is preferred.

Responsibilities

  • Lead engineering on customer engagements, from design through production handover.
  • Build and optimize batch and streaming pipelines using PySpark, SQL, and Delta Lake on Databricks.
  • Implement Medallion architecture across cloud ecosystems and set the layer conventions the team follows.
  • Benchmark Spark jobs, tune SQL, right-size compute, and explain performance and cost trade-offs to customers.
  • Implement data quality and validation as code, including Lakeflow expectations and automated testing in CI.
  • Mentor mid-level and junior engineers through code review, pairing, and design feedback.
  • Partner with the Data Architect on design decisions and escalate when implementation reveals that the design will not hold.
View Full Description & ApplyYou'll be redirected to the employer's site
View details
Apply Now