- Design, implement, and maintain data architecture and ELT/ETL pipelines in Azure Synapse and Azure Machine Learning.
- Migrate source data to Azure Data Lake Storage and normalize entity attributes.
- Establish quality controls, error handling, logging, and validation across all pipelines.
- Integrate source control into analytics codebases to ensure iterative development and architecture stability.
- Develop self-service capabilities for analysts to query and export data.
- Produce comprehensive documentation including data dictionaries and entity relationship diagrams.
- Optimize data ingestion and processing, utilizing columnar formats like Parquet.
PythonSQLPandas+1 more