Senior Databricks Data Engineer
New
J
JobgetherData engineering
Based in BrazilFull-TimeSenior
Salary not disclosed
Apply NowOpens the employer's application page
Job Details
- Experience
- Extensive professional experience delivering and maintaining production-grade data engineering solutions using Databricks.
- Required Skills
- PythonSparkDatabricksPySpark
Requirements
- Extensive professional experience delivering and maintaining production-grade data engineering solutions using Databricks.
- Advanced proficiency in PySpark, Spark SQL, Delta Lake, and Apache Spark.
- Hands-on experience with Delta Live Tables (DLT) / Lakeflow Spark Declarative Pipelines, Change Data Capture (CDC), and Structured Streaming.
- Strong knowledge of Liquid Clustering, Auto Loader, and Delta table optimization techniques.
- Advanced understanding of Unity Catalog, including data governance, security, lineage, and access management.
- Experience designing scalable Lakehouse and Medallion architectures.
- Practical experience using Azure Data Factory for ingestion, orchestration, and system integration.
- Advanced Python skills, including development of REST APIs, microservices, and automation solutions.
- Expertise in Spark performance tuning, including Photon, partitioning, shuffle optimization, and execution plan analysis.
- Experience with Databricks Workflows, CI/CD pipelines, Git, and Declarative Automation Bundles.
- Knowledge of workload optimization, resource sizing, and cloud cost reduction; experience with observability, monitoring, troubleshooting, and production pipeline support.
- Ability to solve complex technical problems and make architectural decisions; experience mentoring and guiding engineering professionals.
- Databricks Certified Data Engineer Associate certification is mandatory; Databricks Certified Data Engineer Professional is strongly preferred.
- Databricks Certified Associate Developer for Apache Spark is nice to have; AWS, Azure, or GCP certifications are advantageous.
Responsibilities
- Design, develop, implement, and improve data engineering solutions using Databricks.
- Define scalable Lakehouse and Medallion architectures across Bronze, Silver, and Gold layers.
- Develop and optimize pipelines using Databricks, PySpark, Spark SQL, Apache Spark, and Delta Lake.
- Implement data ingestion, processing, integration, Change Data Capture, and Structured Streaming solutions.
- Build and manage pipeline orchestration and system integrations using Azure Data Factory.
- Develop REST APIs, microservices, and automation solutions using Python.
- Establish data governance, security, lineage, and access controls through Unity Catalog.
- Optimize performance, scalability, and infrastructure costs, including Photon, partitioning, shuffle management, and execution plan analysis.
- Manage Databricks Workflows, CI/CD, Git repositories, and Declarative Automation Bundles; maintain production observability and support.
- Make architectural decisions, resolve complex technical challenges, mentor engineers, and establish engineering best practices.
View Full Description & ApplyYou'll be redirected to the employer's site
