Senior Data Engineer – Azure Databricks
New
J
JobgetherData engineering
Based in BrazilFull-TimeSenior
Salary not disclosed
Apply NowOpens the employer's application page
Job Details
- Languages
- Advanced English proficiency, both written and spoken
- Experience
- More than eight years of professional experience in data engineering
- Required Skills
- PythonSQLCI/CDAzure DevOpsPySpark
Requirements
- More than eight years of professional experience in data engineering, including the development and maintenance of ETL processes.
- Strong hands-on experience with Azure Databricks and Azure Data Factory (ADF).
- Extensive knowledge of ETL/ELT development, data transformation, and enterprise data integration.
- Advanced SQL skills, including complex query development, optimization, and performance tuning.
- Practical experience with Python and PySpark for data processing and engineering.
- Experience working with Azure Data Lake and integrating data from multiple enterprise systems.
- Experience with data profiling, validation, reconciliation, and quality assurance.
- Experience monitoring, maintaining, and troubleshooting data pipelines in production environments.
- Advanced English proficiency, both written and spoken.
- Experience implementing CI/CD pipelines and DevOps practices is a plus.
- Familiarity with Git, Azure DevOps, or similar version control and deployment tools is a plus.
- Knowledge of cloud monitoring, logging, and observability solutions is a plus.
Responsibilities
- Design, develop, maintain, and optimize data pipelines using Azure Databricks, PySpark, Azure Data Factory, Python, and Azure Data Lake.
- Build and support ETL/ELT processes to integrate enterprise data and meet analytical and business requirements.
- Perform data ingestion, transformation, validation, and reconciliation across multiple source systems.
- Conduct data profiling and quality assessments to identify inconsistencies, resolve issues, and ensure data accuracy and reliability.
- Develop, write, and optimize complex SQL queries to improve data processing performance.
- Tune performance and troubleshoot large-scale data workloads.
- Provide Level 2 production support for incidents involving Azure Data Factory pipelines, Databricks jobs, and data processing workflows.
- Monitor and maintain production pipelines, ensuring availability, operational stability, and efficient resource utilization.
- Support deployment and release processes across development, testing, and production environments.
View Full Description & ApplyYou'll be redirected to the employer's site
