- Define evaluation strategy and success metrics for GenAI systems, aligning offline evaluation with online business and customer experience outcomes.
- Build and scale evaluation frameworks using golden sets, synthetic data, automated regressions, rubric-based grading, and LLM-as-judge where appropriate.
- Apply controls for bias, drift, and reliability in evaluation frameworks.
- Design the data flywheel, including instrumentation, feedback collection, data quality checks, labeling strategy, dataset versioning, and governance.
- Lead cross-functional quality initiatives across product, operations, and engineering.
- Develop and productionize pipelines for dataset creation, model monitoring, evaluation at scale, and continuous pre-deploy and post-deploy testing.
- Drive technical decisions and architecture for evaluation and data infrastructure, balancing speed, rigor, cost, and safety.
Machine LearningA/B testingLLM+1 more