- Build and run production pipelines in Databricks using Lakeflow, Jobs, and dbt, from customer-file ingestion through tables ready for models and reporting.
- Lead the next phases of the Redshift-to-Databricks migration, including landing source tables, moving product analytics onto the lakehouse, and retiring old warehouses.
- Own replication and ingestion from production databases across multiple environments using AWS DMS, Databricks Lakeflow Connect, and Fivetran.
- Set data contracts with internal teams and external ML partners to prevent schema changes from breaking downstream systems.
- Provide product teams with effort estimates and trade-offs, and push back on scope when needed.
- Monitor pipeline health, data quality, and Databricks cost, and participate in the Data on-call rotation.
- Document runbooks, lineage, and design decisions so the team can manage the systems.
- Contribute to the team's use of AI across Engineering, learning and applying technology to solve customer and business problems.
AWSPythonSQL+3 more