Senior Data Engineer - Databricks
New
J
JobgetherData Engineering
USFull-TimeSenior
Salary not disclosed
Apply NowOpens the employer's application page
Job Details
- Experience
- 4+ years of experience in data engineering roles
- Required Skills
- PostgreSQLPythonSQLETLSparkCI/CDDatabricksPySpark
Requirements
- 4+ years of experience in data engineering roles, building and maintaining production-grade data pipelines.
- At least 2 years of hands-on experience with Databricks and the Apache Spark ecosystem across Azure and/or AWS environments.
- Strong proficiency in PySpark, SQL, and Python, with experience delivering scalable data solutions under SLA requirements.
- Hands-on experience with Delta Lake, including schema evolution, ACID transactions, and optimization strategies.
- Experience tuning Databricks workloads, managing compute resources, and improving production pipeline performance.
- Strong knowledge of PostgreSQL, including query optimization and schema design.
- Experience supporting legacy ETL technologies such as SSIS, Informatica, or custom SQL/Python-based workflows.
- Experience working with multi-tenant architectures and enterprise data privacy requirements.
- Solid understanding of data governance, security, compliance, and access management.
- Ability to collaborate effectively with cross-functional stakeholders including data scientists and infrastructure engineers.
- Application of CI/CD best practices for data engineering workflows.
Responsibilities
- Own production support for Databricks-based data platforms, including monitoring, alerting, incident response, and SLA management.
- Design and develop high-performance data pipelines using Databricks, PySpark, SQL, and Python to ingest, transform, and deliver ERP and CRM data at scale.
- Optimize Databricks workloads by improving processing performance, reducing compute costs, and enhancing pipeline efficiency.
- Lead modernization efforts by migrating legacy ETL/ELT workflows into scalable Databricks-based architectures.
- Build and maintain Delta Lake architectures, including schema design, partitioning strategies, and data quality enforcement.
- Implement strong data governance, access controls, and security standards for enterprise data.
- Establish data observability practices to monitor pipeline health and support machine learning outcomes.
- Partner with engineering, product, architecture, and infrastructure teams to ensure seamless integration.
View Full Description & ApplyYou'll be redirected to the employer's site
