Senior Data Engineer - Databricks
New
J
JobgetherData engineering
Based in BrazilFull-TimeSenior
Salary not disclosed
Apply NowOpens the employer's application page
Job Details
- Languages
- Advanced English proficiency
- Required Skills
- PythonSQLGitCI/CDDatabricksPySpark
Requirements
- Demonstrated experience designing, developing, and managing large-scale data pipelines with Databricks, PySpark, Delta Lake, and SQL.
- Strong Python and SQL programming skills for data processing, transformation, and pipeline development.
- In-depth understanding of ETL/ELT frameworks, data warehousing, and distributed data processing.
- Hands-on experience with DataOps practices, including Git version control, CI/CD pipelines, automated testing, and infrastructure as code.
- Familiarity with AWS, Azure, or Google Cloud Platform and their data services.
- Ability to troubleshoot and optimize data solutions for performance, scalability, and reliability.
- Proficiency with Git and collaborative software development workflows.
- Advanced English proficiency.
- Nice to have: experience with data contracts, schema evolution, and compatibility for services and data consumers.
- Nice to have: familiarity with Great Expectations, Soda, dbt tests, or custom data validation solutions.
- Nice to have: experience integrating data solutions with Power BI.
Responsibilities
- Design, develop, maintain, and optimize scalable pipelines using Databricks, PySpark, Delta Lake, and SQL.
- Integrate data from APIs, relational databases, files, and other enterprise systems.
- Build ingestion and transformation processes aligned with the Medallion architecture.
- Collaborate with business and analytics teams to understand requirements and deliver data solutions.
- Ensure data quality, reliability, security, and governance throughout the data lifecycle.
- Apply DataOps practices, including code versioning, automated testing, CI/CD, and infrastructure as code.
- Troubleshoot performance, scalability, and reliability issues in pipelines and distributed processing environments.
- Document processes, technical decisions, standards, and best practices.
- Provide technical guidance and mentorship while collaborating with data scientists, analysts, and platform engineers.
- Contribute to improvements in data architecture, pipeline efficiency, and engineering quality.
View Full Description & ApplyYou'll be redirected to the employer's site
