Data Engineer | Ruby Labs

View full description

Requirements:

• Proficiency in Python for data pipeline development, automation, and tooling.
• Strong SQL skills and experience working with cloud data warehouses (BigQuery preferred).
• Experience with workflow orchestration tools such as Airflow.
• Familiarity with data quality frameworks (e.g., Great Expectations, dbt tests) and anomaly detection methods.
• Experience building monitoring and alerting systems for data pipelines and data quality.
• Ability to write clear, maintainable, and actionable technical documentation.

Responsibilities:

• Develop and maintain ETL/ELT data pipelines to ingest, transform, and deliver data into the data warehouse.
• Design and implement monitoring and alerting systems to proactively detect pipeline failures, anomalies, and data quality issues.
• Establish data quality validation checks and anomaly detection mechanisms to ensure accuracy and trust in data.
• Define and maintain data structures, schemas, and partitioning strategies for efficient and scalable data storage.
• Create and maintain comprehensive documentation of data pipelines, workflows, data models, and data lineage.
• Troubleshoot and resolve issues related to data pipelines, performance, and quality.
• Collaborate with stakeholders to understand data requirements and translate them into reliable engineering solutions.
• Contribute to the continuous improvement of the data platform’s observability, reliability, and maintainability.