Senior Data Engineer (Databricks)
New
J
JobgetherData engineering
This is a fully remote opportunity ... based in Brazil.Full-TimeSenior
Salary not disclosed
Apply NowOpens the employer's application page
Job Details
- Required Skills
- PythonApache KafkaDatabricksPySpark
Requirements
- Have advanced expertise in Databricks, PySpark, Spark SQL, and Delta Lake.
- Bring experience with Delta Live Tables or Lakeflow Spark Declarative Pipelines, CDC, and Structured Streaming.
- Have hands-on knowledge of Liquid Clustering, Auto Loader, and Delta table optimization.
- Understand Unity Catalog, including data governance, security, lineage, and access control.
- Have experience designing and implementing Lakehouse and Medallion architectures.
- Have practical experience with Azure Data Factory for orchestration, ingestion, and integration.
- Have advanced Python skills and experience developing REST APIs, integrations, microservices, and automation.
- Know Spark performance tuning, including Photon, partitioning, shuffle optimization, and execution plan analysis.
- Have experience with Databricks Workflows, CI/CD, Git, and Declarative Automation Bundles.
- Be able to optimize workloads, size compute resources, and reduce processing costs.
- Have experience with observability, monitoring, and production pipeline support.
- Be able to guide other professionals and establish data engineering best practices.
- Relevant certifications include Databricks Certified Data Engineer Associate and Databricks Certified Data Engineer Professional.
Responsibilities
- Define and evolve Databricks data architectures and resolve complex technical challenges.
- Build, maintain, and optimize scalable data ingestion, transformation, and delivery pipelines.
- Design and implement Lakehouse architectures and Bronze, Silver, and Gold Medallion structures.
- Develop batch and streaming processing solutions, including CDC and Structured Streaming.
- Build and orchestrate ingestion and integration processes using Azure Data Factory, Auto Loader, and Lakeflow.
- Implement data governance, security, lineage, and access controls using Unity Catalog.
- Establish CI/CD, version control, and automated deployments using Git and Databricks Workflows.
- Monitor production pipelines and optimize Spark performance, compute resources, and processing costs.
- Mentor professionals and promote data engineering standards and best practices.
View Full Description & ApplyYou'll be redirected to the employer's site
