- Own and maintain the methodology G2 uses to evaluate AI agents, and defend its standards to internal and external audiences.
- Work with engineers and product managers to build and improve evaluations behind current offerings, scope features, and turn methods into production-ready components.
- Build environments and task sets that reflect business work given to AI agents and remain useful as models improve.
- Develop scoring approaches using automated checks, LLM-based judges, and human review; measure consistency and reliability.
- Audit tasks and graders, quantify noise, and establish when differences between agents are meaningful.
- Present and defend evaluation findings to buyers, customers, partners, and the research community.
- Mentor researchers and engineers, review evaluation methodology, and document repeatable standards.
- Choose and pursue research problems, including agent simulation, LLM-as-judge reliability, multimodal evaluation, long-horizon workflows, and test-set contamination.
- Track research and benchmark developments and translate relevant ideas into product and roadmap decisions.
Machine Learning