Lead Data Engineer, Databricks
V
V4CData engineering
United StatesFull-TimeLead
Salary not disclosed
Apply NowOpens the employer's application page
Job Details
- Experience
- 7+ years of data engineering plus experience leading projects or teams
- Required Skills
- AWSPythonSQLAzureSparkCI/CDDatabricksPySpark
Requirements
- Have 7+ years of data engineering experience and experience leading projects or teams.
- Demonstrate strong proficiency in Python, SQL, Apache Spark, and Delta Lake.
- Have production experience building batch and streaming pipelines at enterprise scale.
- Have hands-on experience with AWS, Azure, or GCP and its data services.
- Have experience with data modeling, Git, and CI/CD for data workloads.
- Demonstrate experience optimizing Spark performance and managing platform cost.
- Communicate technical trade-offs to non-technical stakeholders.
- Healthcare or life sciences experience is preferred but not required.
- Databricks Certified Data Engineer Associate or Professional certification is preferred.
- Experience with Unity Catalog, Lakeflow, or Databricks Asset Bundles is preferred.
- Experience with dbt, Airflow, or equivalent orchestration and transformation tooling is preferred.
- Experience migrating a legacy ETL estate to a modern cloud platform, or a consulting or client-facing engineering background, is preferred.
Responsibilities
- Lead engineering on customer engagements from design through production handover.
- Build and optimize batch and streaming pipelines using PySpark, SQL, and Delta Lake on Databricks.
- Implement Medallion architecture across cloud ecosystems and establish layer conventions for the team.
- Benchmark Spark jobs, tune SQL, right-size compute, and explain performance and cost trade-offs to customers.
- Implement data quality and validation as code, including Lakeflow expectations and automated testing in CI.
- Mentor mid-level and junior engineers through code review, pairing, and design feedback.
- Partner with the Data Architect on design decisions and escalate when implementation challenges the design.
View Full Description & ApplyYou'll be redirected to the employer's site