Job Details
- Languages
- English
- Experience
- 9+ years
- Required Skills
- PythonSQLApache AirflowETLGCPBigQuery
Requirements
- At least 9+ years experience as DE - Python, GCP, BigQuery
- Object oriented programming, Python and SQL
- Strong knowledge in cloud computing platforms - Google Cloud
- Ability to design, build, and deploy data pipelines in the cloud
- Experience in composer or Apache Airflow and knowledge of Dataplex is a plus
- Experience in dq checks implementation, any frameworks like Clouddq, Pydeequ
- Good Knowledge of Dq dimensions
- Experience working with GCP cloud-based infrastructure & systems
- Programming skills (SQL, Python, other scripting)
- Proficient in data modelling techniques and database optimization
- Knowledge of query optimization, indexing, and performance tuning
- Proficient in database management systems such as SQL (Big Query is a must), NoSQL
- Ability to design, configure, and manage databases
- Experience with data integration tools and techniques, such as ETL and ELT
Responsibilities
- Modelling data from various sources and technologies.
- Troubleshooting and supporting complex problems to deliver new features.
- Designing and optimizing data storage architectures (data lakes, data warehouses, distributed file systems).
- Implementing techniques like partitioning, compression, or indexing.
- Identifying and resolving bottlenecks, tuning queries, and implementing caching strategies.
- Identifying and resolving issues related to data processing, storage, or infrastructure.
- Monitoring system performance, identifying anomalies, and conducting root cause analysis.
- Train and mentor junior data engineers.
