Engenheiro de Dados Sênior GCP/DBT
New
BrazilFull-TimeSenior
Salary not disclosed
Apply NowOpens the employer's application page
Job Details
- Experience
- Minimum 3 years
- Required Skills
- PythonSQLGCPGitTerraformBigQuerydbtPySpark
Requirements
- Minimum 3 years of hands-on experience with Google Cloud Platform (GCP).
- Strong expertise in BigQuery, including data modeling, performance optimization, partitioning, clustering, and security.
- Solid experience with Cloud Storage, including bucket management, lifecycle policies, and IAM controls.
- Experience with Dataproc (Spark/Hadoop), including cluster provisioning, tuning, and job optimization.
- Strong experience with DBT (minimum 3 years required) and data transformation workflows.
- Advanced knowledge of PySpark and Python for data pipelines and automation.
- Advanced SQL skills for analytical modeling and transformations.
- Experience with orchestration tools such as Dataflow or Composer.
- Experience with Git and GitFlow-based version control practices.
- Knowledge of cloud security concepts, including IAM, VPC, networking, and access control.
- Ability to work with shell scripting for automation tasks.
- Experience working in Agile environments and using Jira for task management.
Responsibilities
- Design, plan, and evolve data architectures on Google Cloud Platform, ensuring scalability, performance, and cost efficiency.
- Analyze and define data pipelines and workloads across GCP services such as BigQuery, Cloud Storage, Dataproc, Dataflow, and Composer.
- Develop and maintain ELT/ETL pipelines using DBT, PySpark, and related data processing frameworks.
- Model data structures in BigQuery, including partitioning and clustering strategies for optimized performance.
- Implement and manage data lake architectures using Bronze, Silver, and Gold layers in Cloud Storage.
- Provision and manage cloud infrastructure using Terraform, ensuring secure and standardized environments.
- Optimize BigQuery queries and Spark jobs to improve performance and reduce operational costs.
- Implement data validation, quality controls, and monitoring mechanisms across pipelines.
- Manage security, IAM policies, networking configurations, and governance best practices in GCP.
- Collaborate with stakeholders in Agile environments, ensuring clear communication and alignment on technical solutions.
View Full Description & ApplyYou'll be redirected to the employer's site
