Senior Databricks Data Engineer
New
J
JobgetherData Engineering
BrazilFull-TimeSenior
Salary not disclosed
Apply NowOpens the employer's application page
Job Details
- Experience
- 5+ years
- Required Skills
- AWSPythonETLGitCI/CDDatabricksPySpark
Requirements
- 5+ years of experience in Data Engineering roles.
- Advanced proficiency in PySpark and Python, including large-scale batch pipeline development, testing practices, and engineering standards.
- Strong SQL expertise and experience with complex transformations, performance tuning, and relational or dimensional data modeling.
- Hands-on experience with Databricks and Delta Lake, including production workflows, jobs, Medallion architecture, Delta Live Tables, Lakeflow, and Asset Bundles.
- Experience migrating or rebuilding ETL pipelines by translating business rules from legacy systems into Spark-based solutions with CDC and batch ingestion patterns.
- Knowledge of AWS data services such as S3, Glue, EMR, Athena, Lambda, DMS, and Step Functions.
- Experience with Azure Synapse environments, including SQL pools and pipelines, to support reverse engineering activities.
- Strong understanding of data quality practices, automated testing, data quality gates, and legacy versus new platform reconciliation.
- Experience using Git and CI/CD practices for data pipeline development.
Responsibilities
- Rebuild legacy data warehouse pipelines using Databricks, PySpark, and Spark SQL based on specifications created through reverse engineering.
- Implement bronze, silver, and gold data layers following the Medallion architecture, ingestion standards, and project reconstruction guidelines.
- Execute migration waves by business domain while maintaining coexistence between legacy systems and the new platform until final cutover.
- Develop business logic transformations and implement automated testing across data pipelines.
- Perform data reconciliation and validate parity between legacy data warehouse outputs and new Lakehouse implementations.
- Optimize pipeline performance and cloud costs through techniques such as partitioning, OPTIMIZE/Z-ORDER strategies, and job sizing improvements.
- Contribute to technical documentation, migrated business rules, and prioritization of future migration activities.
- Collaborate with engineering teams to improve data quality, reliability, and delivery processes.
View Full Description & ApplyYou'll be redirected to the employer's site
