Senior Data Developer
New
C
CI&TData platforms
BrazilFull-TimeSenior
Salary not disclosed
Apply NowOpens the employer's application page
Job Details
- Languages
- Advanced english
- Required Skills
- AWSPythonSQLGCPAzureSparkCI/CDDatabricks
Requirements
- Have solid experience building data pipelines in production environments, including incident response and troubleshooting.
- Be proficient in Python and SQL for building and maintaining ETL/ELT solutions.
- Have experience with large-scale distributed data processing using Databricks and Spark.
- Have hands-on experience with at least one major cloud platform: Azure, AWS, or GCP.
- Know Lakehouse architecture, including Raw, Trusted, Refined, and Analytics layers, and Medallion architecture.
- Know data quality practices, monitoring, and troubleshooting in data environments.
- Be familiar with version control, CI/CD, and software engineering best practices applied to data.
- Have advanced English.
Responsibilities
- Develop and maintain data pipelines for the Raw, Trusted, Refined, and Analytics layers.
- Drive the evolution of the Medallion architecture and Unity Catalog.
- Implement integrations with Databricks and ADLS.
- Ingest and standardize data from APIs, databases, files, and other sources.
- Implement observability sensors to monitor platform health, performance, and integrity.
- Investigate ingestion and processing failures using logs, metrics, and targeted testing.
- Respond to production incidents and implement fixes and improvements to reduce recurrence.
- Implement data quality controls and maintain data refresh routines.
- Optimize pipelines and processing resource use to support performance and cost improvements.
- Document technical solutions and architecture decisions, and share knowledge through code reviews and technical sessions.
View Full Description & ApplyYou'll be redirected to the employer's site
