Lead Data Software Engineer
New
J
JobgetherData engineering
Based in BrazilFull-TimeLead
Salary not disclosed
Apply NowOpens the employer's application page
Job Details
- Languages
- Upper-intermediate English proficiency or higher (B2).
- Experience
- At least 5 years of relevant professional experience. At least 1 year of experience leading and managing development teams.
- Required Skills
- PySpark
Requirements
- At least 5 years of relevant professional experience.
- At least 1 year of experience leading and managing development teams.
- Strong experience with PySpark and Spark SQL for large-scale data processing workflows.
- Hands-on experience developing and managing Databricks Lakeflow pipelines and Jobs for data orchestration.
- Proficiency with Auto Loader and JDBC/CDC ingestion strategies, including integration with DB2 and SQL Server.
- Strong knowledge of Delta Lake and Unity Catalog for data storage, management, and governance.
- Experience implementing data quality controls and source reconciliation processes.
- Experience writing unit tests with pytest.
- Working knowledge of Git and Databricks Asset Bundles for version control and deployment management.
- Upper-intermediate English proficiency or higher (B2).
- Experience participating in DataStage migration projects.
- Experience extracting and integrating data from Dynamics 365 or Dataverse.
- Databricks certification.
Responsibilities
- Design, build, and maintain scalable data pipelines that ensure reliable data flow across multiple systems.
- Develop and manage ingestion pipelines that collect and process data from diverse source systems.
- Write clean, maintainable, high-quality, thoroughly tested code based on software engineering principles.
- Collaborate with cross-functional teams to translate business and data requirements into technical solutions.
- Monitor pipeline performance, identify potential issues, and maintain system reliability.
- Investigate and resolve data quality issues to ensure dataset accuracy, consistency, and reliability.
- Participate in code reviews and uphold engineering quality standards.
- Create and maintain technical documentation for knowledge sharing and operational continuity.
- Participate in Agile ceremonies, including sprint planning and regular team meetings.
- Provide technical leadership and support the development team in delivering data engineering solutions.
View Full Description & ApplyYou'll be redirected to the employer's site
