Principal Research Scientist

G2AI agent evaluation

Remote (US)Full-TimePrincipal

Salary$190K - $235K; $190K – $235K • Offers Equity

Apply NowOpens the employer's application page

Job Details

Experience: 5+ years building or evaluating ML systems, including hands-on work with LLMs or LLM-based agents. 2+ years experience managing small to medium teams preferred.
Required Skills: Machine Learning

Experience working with evaluation design or dataset curation in multi-agentic systems.
5+ years building or evaluating ML systems, including hands-on work with LLMs or LLM-based agents.
A graduate degree (MS or PhD) in machine learning, NLP, computer science, statistics, or a related field, or equivalent applied research experience.
A track record of original machine learning ideas or clear improvements, demonstrated through first-author publications or projects led end-to-end.
Ability to own a research agenda, select problems to solve, and carry long-running projects forward independently.
A track record of shipping ML work in a product setting and working closely with engineers and product managers.
Experience setting technical direction, mentoring researchers or engineers, and reviewing methodology.
Familiarity with AI benchmarking and how benchmark scores can mislead.
Sound statistical judgment in experimental design, variance, significance, and agreement between human and automated raters.
Ability to make practical trade-offs that serve customer needs.
Clear writing and ability to explain methodology to non-specialists and defend it to skeptical audiences.
2+ years managing small to medium teams preferred.

Own and maintain the methodology G2 uses to evaluate AI agents, and defend its standards to internal and external audiences.
Work with engineers and product managers to build and improve evaluations behind current offerings, scope features, and turn methods into production-ready components.
Build environments and task sets that reflect business work given to AI agents and remain useful as models improve.
Develop scoring approaches using automated checks, LLM-based judges, and human review; measure consistency and reliability.
Audit tasks and graders, quantify noise, and establish when differences between agents are meaningful.
Present and defend evaluation findings to buyers, customers, partners, and the research community.
Mentor researchers and engineers, review evaluation methodology, and document repeatable standards.
Choose and pursue research problems, including agent simulation, LLM-as-judge reliability, multimodal evaluation, long-horizon workflows, and test-set contamination.
Track research and benchmark developments and translate relevant ideas into product and roadmap decisions.

View Full Description & ApplyYou'll be redirected to the employer's site