Tech Lead Data - Databricks
New
J
JobgetherData engineering
Based in BrazilFull-TimeLead
Salary not disclosed
Apply NowOpens the employer's application page
Job Details
- Languages
- Advanced English proficiency
- Required Skills
- PythonSQLGitAzureDatabricksPySpark
Requirements
- Have strong experience in Data Engineering projects.
- Have previous experience providing technical and operational leadership as a Tech Lead or Senior Engineer.
- Bring hands-on experience with Databricks, PySpark, and Spark SQL, particularly for batch data pipelines; streaming knowledge is needed when required.
- Have deep knowledge of Delta Lake and practical experience implementing the Medallion architecture across Bronze, Silver, and Gold layers.
- Have experience with Azure data services, including Azure Databricks, Azure Data Lake Storage (ADLS), and identity and security services such as Azure Active Directory.
- Have strong SQL and Python skills.
- Understand dimensional data modeling, consumption-oriented models, and schema and data contract best practices.
- Have experience with DataOps practices including Git, CI/CD for data pipelines, automated testing, and technical documentation.
- Be able to review architecture and code, troubleshoot performance and scalability issues, and guide engineering teams toward solutions.
- Have advanced English proficiency for stakeholder collaboration and technical documentation.
Responsibilities
- Provide technical leadership to Data Engineering teams through solution design, code reviews, technical decisions, and blocker resolution.
- Collaborate with the Data Architect to define technical vision, coding standards, data modeling conventions, and Lakehouse best practices.
- Design, develop, and oversee Databricks pipelines with PySpark and Spark SQL across Bronze, Silver, and Gold layers.
- Establish DataOps practices for Git version control, CI/CD, automated testing, data quality validation, and documentation.
- Optimize pipeline performance through partitioning, Spark tuning, cloud cost management, monitoring, observability, and alerting.
- Ensure access control, security, and data policies across Azure and Databricks.
- Partner with business and analytics stakeholders to translate requirements into technical solutions and delivery plans.
- Mentor engineers and develop reusable templates, playbooks, and technical references.
View Full Description & ApplyYou'll be redirected to the employer's site
