Senior Data Developer (AI / Databricks)
New
J
JobgetherData engineering
The position is remote within BrazilFull-TimeSenior
Salary not disclosed
Apply NowOpens the employer's application page
Job Details
- Experience
- Substantial professional experience in senior-level data engineering or data development.
- Required Skills
- PythonSQLMicrosoft AzureCI/CDDatabricksPySpark
Requirements
- Bring substantial professional experience in senior-level data engineering or data development, including designing and maintaining production-grade data pipelines.
- Have strong experience with Microsoft Azure data services and Databricks, including distributed data processing solutions.
- Be proficient in PySpark, Python, and SQL for data transformation, analysis, integration, and automation.
- Know ETL/ELT processes and data pipeline architecture, including ingestion, transformation, orchestration, and delivery.
- Have experience with Azure Data Factory, Azure Synapse Analytics, Azure Data Lake Storage, and Apache Spark.
- Have practical knowledge of CI/CD practices and automated deployment workflows.
- Have experience with Git and GitHub, and familiarity with Azure DevOps for versioning and deployment automation.
- Have experience in conceptual, logical, and physical data modeling, including performance optimization, scalability, and data governance.
- Have experience integrating large datasets and maintaining data consistency and quality across complex workflows.
- Be familiar with cloud migration projects and secure, reliable data transfers.
- Be able to collaborate with business and data science teams on AI model deployment and cloud-based data solutions.
- Have analytical and problem-solving skills, including investigating data issues, identifying root causes, and implementing improvements.
- Have technical documentation and communication skills for collaboration across multidisciplinary teams.
- Preferred: experience with AI projects, automated data pipelines, Master Data Management initiatives, Databricks MLflow, Data Lake and Data Warehouse architectures, or Azure Machine Learning or similar tools.
Responsibilities
- Design, develop, maintain, and optimize ETL/ELT data pipelines using Microsoft Azure, Databricks, and PySpark.
- Implement automated workflows for data ingestion, transformation, integration, processing, and delivery to downstream systems and applications.
- Process customer registration and master data, ensuring accuracy, consistency, integrity, and availability.
- Develop and maintain CI/CD pipelines and manage source code and collaborative development workflows using GitHub and Azure DevOps.
- Design and optimize conceptual, logical, and physical data models for scalability, performance, and data governance.
- Build and evolve cloud-based data solutions using Azure Data Factory, Azure Synapse Analytics, Azure Data Lake Storage, and Spark.
- Contribute to cloud migrations, maintaining secure data transfer, integrity, quality, and continuity.
- Collaborate with business stakeholders, data scientists, and engineering teams to integrate data solutions and deploy AI models in cloud environments.
- Establish data quality controls, monitor pipeline performance, and improve data processing workflows.
- Investigate data issues, document architectures and processes, and contribute to data architecture, governance, and AI-enabled workflows.
View Full Description & ApplyYou'll be redirected to the employer's site
