- Operationalize model training, evaluation, packaging, and deployment using Databricks, Delta Lake, and medallion architecture.
- Implement Unity Catalog model governance, lineage tracking, and access control.
- Develop reusable job templates, cluster policies, and standardized deployment patterns.
- Deploy and manage ML and GenAI solutions including risk scoring, anomaly detection, predictive maintenance, NLP, and RAG pipelines.
- Build and optimize LLM pipelines using vector databases, model serving endpoints, and inference workflows.
- Optimize models using quantization, caching, and performance tuning techniques.
- Implement batch and real-time inference pipelines with defined SLAs.
- Implement data contracts, schema validation, and data quality checks across ML pipelines.
- Ensure secure handling of sensitive data including PII detection, classification, and obfuscation.
- Maintain full lineage from data sources to deployed models and serving endpoints.
- Enforce data residency, governance, and compliance policies.
- Implement CI/CD pipelines using GitHub Actions and Databricks Asset Bundles.
- Automate deployments across DEV, QA, and PROD environments.
- Develop unit and integration tests for data pipelines and ML models.
- Ensure version control, reproducibility, and automated deployment workflows.
- Monitor pipeline health, model performance, drift, and system reliability.
- Implement alerting, incident response workflows, and automated ticketing.
- Track LLM performance metrics including latency, hallucination rates, and API costs.
- Develop runbooks, disaster recovery procedures, and operational documentation.
- Apply tagging policies and cost tracking for ML infrastructure.
- Support budget monitoring, cost optimization, and resource management.
AWSPythonSQL+6 more