Senior Data Engineer - Databricks
New
US - Remote, multiple time zonesFull-TimeSenior
Salary155,000 - 185,000 USD per year
Apply NowOpens the employer's application page
Job Details
- Experience
- 4+ years of data engineering experience
- Required Skills
- AWSPostgreSQLPythonSQLETLAzureSparkDatabricksPySpark
Requirements
- 4+ years of data engineering experience.
- At least 2 years on Databricks or the Apache Spark ecosystem across Azure and/or AWS.
- Proficiency in PySpark, SQL, and Python with experience building and operating production-grade pipelines.
- Hands-on experience with Delta Lake including schema evolution, ACID transactions, optimize/vacuum lifecycle, and incremental/streaming processing.
- Hands-on experience with pipeline performance tuning and compute optimization in production Databricks environments.
- Solid working knowledge of PostgreSQL including query optimization and schema design.
- Experience supporting and maintaining legacy ETL tooling (SSIS, Informatica, custom Python/SQL pipelines).
- Experience supporting large-scale multi-tenant architectures with a focus on isolation and data privacy.
- Proven ability to work collaboratively across data science, product, and infrastructure teams.
- Strong understanding of data governance, security, and compliance principles.
Responsibilities
- Own Databricks production support for the Sugar Predict data platform, including monitoring, alerting, and incident response across all production data flows.
- Maintain and report on SLA performance metrics for data pipeline delivery, ensuring visibility into platform health and accountability.
- Identify and implement pipeline optimizations that reduce Databricks compute costs, improve throughput, and reduce processing windows.
- Migrate legacy ETL/ELT pipelines to Databricks, building automation tooling to reduce manual intervention.
- Support new customer onboarding by provisioning, validating, and hardening tenant data pipelines.
- Design and build high-performance Databricks pipelines that ingest, transform, and serve ERP and CRM data at scale across Azure and AWS.
- Own the Delta Lake architecture including schema design, partitioning, data quality enforcement, and incremental processing.
- Enforce data security best practices, including role-based access control and secrets management.
- Support a globally distributed operation through on-call rotation and after-hours incident response.
View Full Description & ApplyYou'll be redirected to the employer's site
