GCP Data Engineer

View full description

Requirements:

• 4+ years of experience with PySpark and SQL for building scalable ETL pipelines.
• Strong proficiency in Python programming.
• Knowledge of GCP Data Analytics ecosystem (BigQuery, PySpark, SQL, etc.).
• Experience with Airflow/Composer for workflow orchestration.
• Experience with in-memory applications, database design, and data integration.
• Strong analytical thinking and problem-solving abilities.

Responsibilities:

• Design, build, and maintain scalable and robust ETL/ELT pipelines using PySpark and SQL.
• Work on data extraction, transformation, and loading processes from multiple sources into data warehouses such as BigQuery.
• Leverage GCP data analytics tools (BigQuery, DataProc, Cloud Functions, etc.) to process and analyze data.
• Optimize data workflows for benchmarking, performance, and tuning to ensure efficiency and reliability.
• Collaborate with engineering and analytics teams to develop data integration solutions that meet business needs.
• Ensure the accuracy and quality of data by implementing strong in-memory applications and database designs.
• Implement monitoring and alerting for pipelines and workflows to ensure data consistency and issue resolution.