- Design and build end-to-end data and AI pipelines using Azure Databricks.
- Develop robust ETL/ELT workflows using Python (PySpark) and SQL.
- Implement CI/CD pipelines for Databricks deployments including jobs and workflows.
- Integrate Databricks with Azure services like Data Lake, Key Vault, and Azure OpenAI.
- Develop and deploy ML and Generative AI models including LLMs and RAG pipelines.
- Collaborate with data scientists and stakeholders to move models into production.
- Optimize production jobs for performance, cost, and reliability.
- Implement logging, monitoring, and AI safety guardrails.
PythonSQLMLFlow+4 more