Senior Data Engineer - Databricks
New
Based in the United StatesFull-TimeSenior
SalaryCompetitive compensation package with performance-based incentives
Apply NowOpens the employer's application page
Job Details
- Experience
- 4+ years of experience in data engineering
- Required Skills
- AWSPostgreSQLPythonSQLETLAzureSparkDatabricksPySpark
Requirements
- 4+ years of experience in data engineering with strong exposure to production-grade data pipelines.
- At least 2+ years of hands-on experience with Databricks and/or Apache Spark ecosystems across Azure and/or AWS.
- Strong proficiency in Python, PySpark, and SQL for building scalable data transformations and workflows.
- Deep experience with Delta Lake, including schema evolution, ACID transactions, and incremental/streaming processing.
- Proven expertise in performance tuning and optimizing Databricks workloads in production environments.
- Solid understanding of PostgreSQL and relational database design, including query optimization and integration patterns.
- Experience working with legacy ETL systems (e.g., SSIS, Informatica, or custom pipelines) and modernization efforts.
- Familiarity with multi-tenant architectures, data isolation strategies, and enterprise-grade data governance.
- Strong collaboration skills with cross-functional teams including data science, product, and infrastructure.
- Knowledge of CI/CD practices, version control, and automated testing for data engineering workflows.
Responsibilities
- Own the design, development, and production support of Databricks-based data pipelines powering ERP and CRM data ingestion, transformation, and delivery at scale.
- Monitor and optimize production data workflows, ensuring SLA compliance, system reliability, and rapid incident resolution across distributed environments.
- Lead performance tuning and cost optimization initiatives across Databricks workloads, improving throughput and reducing compute costs.
- Drive migration of legacy ETL/ELT systems to modern Databricks architecture, ensuring seamless and automated transitions.
- Architect and maintain Delta Lake structures, including schema design, partitioning strategies, and incremental processing frameworks.
- Implement robust data quality, observability, and monitoring systems to ensure integrity across pipelines and ML inputs.
- Support multi-tenant data onboarding, ensuring secure, isolated, and validated pipelines for enterprise customers.
- Collaborate cross-functionally with ML, product, and enterprise architecture teams to align data infrastructure with platform needs.
- Maintain operational readiness through documentation, CI/CD practices, runbooks, and participation in on-call support rotations.
View Full Description & ApplyYou'll be redirected to the employer's site
