Lead Data Engineer
New
J
JobgetherData engineering
Based in BrazilFull-TimeLead
Salary not disclosed
Apply NowOpens the employer's application page
Job Details
- Languages
- Upper-intermediate English proficiency or higher (B1 stated as the minimum in the source).
- Experience
- At least 5 years of relevant professional experience; at least 1 year of experience leading and managing development teams.
- Required Skills
- AWSPythonApache AirflowPandasCI/CDTerraformGitHub ActionsPySpark
Requirements
- At least 5 years of relevant professional experience.
- At least 1 year of experience leading and managing development teams.
- Strong Python programming skills applied to data engineering.
- Practical experience using Pandas for data manipulation and analysis.
- Advanced working knowledge of PySpark for large-scale distributed data processing.
- Experience with AWS MWAA or Apache Airflow for workflow orchestration.
- Familiarity with AWS Athena for querying and analyzing data stored in AWS.
- Hands-on experience with Terraform for infrastructure automation and management.
- Working knowledge of GitHub Actions and automated CI/CD pipelines.
- Good understanding of DevOps principles and their application to data engineering.
- Strong data engineering, troubleshooting, and complex problem-solving skills.
- Upper-intermediate English proficiency or higher (B1 stated as the minimum in the source).
- Nice to have: experience with GitHub Copilot, Agile or Scrum, financial services or investment management, and Generative AI applications and SDLC practices.
Responsibilities
- Design, develop, and maintain scalable data pipelines running on AWS infrastructure.
- Build data-processing solutions using Python, Pandas, and PySpark.
- Configure and manage workflows using AWS MWAA or Apache Airflow.
- Develop and optimize queries with AWS Athena.
- Automate infrastructure provisioning and deployment using Terraform and GitHub Actions.
- Implement testing, monitoring, troubleshooting, and data-quality controls.
- Participate in code reviews and promote engineering best practices.
- Collaborate with engineering, product, and business stakeholders in an Agile environment.
- Provide technical guidance and leadership to support the development team's delivery and performance.
View Full Description & ApplyYou'll be redirected to the employer's site
