Mid Data Developer (Azure + Databricks)
New
J
JobgetherData Engineering
BrazilFull-TimeMiddle
Salary not disclosed
Apply NowOpens the employer's application page
Job Details
- Required Skills
- SQLETLGitMicrosoft SQL ServerAzureSparkCI/CDDatabricksPySpark
Requirements
- Proven experience in data engineering, with hands-on experience developing and maintaining data pipelines.
- Experience working in production environments, including monitoring, troubleshooting, incident investigation, and continuous improvement.
- Experience handling large volumes of data and developing scalable processing solutions.
- Experience participating in data migration, modernization, or data lake development projects.
- Strong experience with cloud-based data environments, particularly Azure.
- Practical knowledge of Databricks for distributed data processing.
- Proficiency with Apache Spark and PySpark, including data transformation, processing optimization, and partitioning.
- Experience with SQL and SQL Server, as well as knowledge of Delta Lake.
- Familiarity with Git and CI/CD practices and DevOps principles.
- Strong analytical and problem-solving skills for incident investigation and root cause analysis.
Responsibilities
- Design, develop, maintain, and optimize robust ETL/ELT data pipelines, applying best practices for low latency, resilience, data quality, observability, and reliable data processing.
- Develop scalable data processing workflows using Apache Spark, PySpark, and Databricks, applying efficient partitioning strategies and cost optimization techniques for large-scale workloads.
- Structure, organize, and manage data using SQL Server and Delta Lake, contributing to reliable and scalable storage architectures for modern data platforms.
- Apply DevOps practices across data engineering workflows, including Git-based version control, CI/CD automation, deployment routines, and continuous monitoring of production processes.
- Investigate production incidents, identify root causes, implement corrective actions, and contribute to the continuous improvement of data pipelines and production routines.
- Contribute to data platform modernization initiatives, supporting the development of scalable, product-oriented architectures and helping evolve data engineering practices.
View Full Description & ApplyYou'll be redirected to the employer's site
