Data Reliability Engineer

View full description

Requirements:

• 5+ years of hands-on experience in supporting data engineering teams, strongly emphasizing data pipeline enhancement and optimization, and data integration.
• Proficient in cloud computing, preferably Google Cloud Platform (GCP), but AWS and Azure are also valid.
• Experience with cloud data-related services such as BigQuery, Dataflow, Cloud Composer, Dataproc, Cloud Storage, Pub/Sub, or the correlated services from other providers.
• Solid proficiency with Python in terms of data processing.
• Knowledge of SQL and experience with relational databases.
• Proven experience optimizing data pipelines toward efficiency, reducing operational costs, and reducing the number of issues/failures.
• Solid knowledge of monitoring, troubleshooting, and resolving data pipeline issues.
• Familiarity with version control systems like Git.

Responsibilities:

• Design and implement scalable data pipeline architectures in collaboration with Data Engineers.
• Continuously optimize data pipeline efficiency to reduce operational costs and minimize issues and failures.
• Monitor performance and reliability of data pipelines, enhancing reliability through data quality, analysis, and testing.
• Build and manage automated alerting systems for data pipeline issues.
• Automate repetitive tasks in data processing and management.
• Develop and manage disaster recovery and backup plans.
• In collaboration with other Data Engineering teams, conduct capacity planning for data storage and processing needs.
• Develop and maintain comprehensive documentation for data pipeline systems and processes, and provide knowledge transfer to data-related teams.
• Monitor, troubleshoot and resolve production issues in data processing workflows.
• Maintain infrastructure reliability for data pipelines, enterprise datahub, HPBI, and MDM systems.
• Conduct post-incident reviews and implement improvements for data pipelines.