- Design, build, and maintain reliable, scalable ETL/ELT pipelines for ingesting, transforming, and delivering data.
- Develop Apache Airflow DAGs to orchestrate pipeline tasks, schedules, dependencies, and data-processing workflows.
- Integrate data from databases, APIs, and files, applying transformations and preserving data consistency.
- Develop SQL transformations and Snowflake database structures, including schemas, tables, and views.
- Optimize SQL queries and Snowflake workloads by evaluating query performance, warehouse sizing, utilization, and cost.
- Monitor workflow runs, investigate failures, and implement corrective actions, alerts, and operational logging.
- Develop Python scripts and automation to support pipeline operations.
- Develop and maintain automated data validation checks and tests, and investigate data-quality issues.
- Document data lineage, transformation logic, workflow dependencies, recovery steps, and troubleshooting guidance.
- Collaborate with technical teams and stakeholders, participate in code reviews and design discussions, and contribute to version control, automated testing, and CI/CD practices.
PythonSQLApache Airflow+4 more