Lead Data Engineer, Databricks
New
V
V4CData engineering consulting
Remote, US; Source API remote eligibility restrictions: United StatesFull-TimeLead
Salary not disclosed
Apply NowOpens the employer's application page
Job Details
- Experience
- 7+ years of data engineering plus experience leading projects or teams
- Required Skills
- AWSPythonSQLGCPAzureSparkCI/CDDatabricksPySpark
Requirements
- Have 7+ years of data engineering experience and experience leading projects or teams.
- Demonstrate strong proficiency in Python, SQL, Apache Spark, and Delta Lake.
- Have production experience building both batch and streaming pipelines at enterprise scale.
- Have hands-on experience with AWS, Azure, or GCP and its data services.
- Have experience with data modeling, Git, and CI/CD for data workloads.
- Demonstrate experience optimizing Spark performance and managing platform cost.
- Have strong client communication skills, including the ability to explain technical trade-offs to non-technical stakeholders.
- Healthcare or life sciences experience is preferred but not required.
- Databricks Certified Data Engineer Associate or Professional certification is preferred.
- Hands-on experience with Unity Catalog, Lakeflow, or Databricks Asset Bundles is preferred.
- Experience with dbt, Airflow, or equivalent orchestration and transformation tooling is preferred.
- Experience migrating a legacy ETL estate to a modern cloud platform is preferred.
- Consulting or client-facing engineering experience is preferred.
Responsibilities
- Lead engineering on customer engagements, from design through production handover.
- Build and optimize batch and streaming pipelines using PySpark, SQL, and Delta Lake on Databricks.
- Implement Medallion architecture across cloud ecosystems and set the layer conventions the team follows.
- Benchmark Spark jobs, tune SQL, right-size compute, and explain performance and cost trade-offs to customers.
- Implement data quality and validation as code, including Lakeflow expectations and automated testing in CI.
- Mentor mid-level and junior engineers through code review, pairing, and design feedback.
- Partner with the Data Architect on design decisions and escalate when implementation reveals that the design will not hold.
View Full Description & ApplyYou'll be redirected to the employer's site