Director, Research - AI Evals
New
F
FigmaSoftware Design
This is a full time role that can be held from one of our US hubs or remotely in the United States.Full-TimeDirector
Salary$258,000 — $348,000 USD
Apply NowOpens the employer's application page
Job Details
- Experience
- 10+ years
- Required Skills
- Data AnalysisRegression testing
Requirements
- 10+ years of experience in product, research, applied research, or a closely related field
- 2+ years of management experience
- Direct, hands-on experience owning the evaluation of AI/LLM-powered products
- Expertise designing and running AI evaluation, including human evaluation programs, rubric and benchmark/golden-dataset construction, and inter-rater reliability
- Sound judgment about when and how to apply automated/model-based approaches (e.g., LLM-as-judge)
- Strength across both qualitative and quantitative methods, comfort with data and metrics, and the ability to reason about model behavior
- Demonstrated success in identifying the riskiest assumptions behind an ambiguous quality question, prioritizing them, and designing right-sized evaluation to build confidence
- Proven track record of gaining buy-in from executive and cross-disciplinary stakeholders
Responsibilities
- Own AI evaluation methods and operations for Figma's AI-powered experiences — define quality dimensions, design how we measure them, and turn results into decision-ready signal
- Build and maintain evaluation frameworks, rubrics, golden datasets, and quality bars, combining human evaluation with automated/model-based approaches (e.g., LLM-as-judge) where appropriate
- Partner with engineering to stand up repeatable, reproducible evaluation pipelines and regression testing, so evaluation is a routine part of how AI features are built and shipped
- Produce clear readouts and dashboards that let stakeholders confidently make go/no-go and prioritization decisions
- Socialize a shared definition of quality so evaluation standards are adopted across teams rather than re-invented — and advocate for evaluation as a strategic partner in the product process
- Manage a small team to execute our AI evals in partnership with contractors, internal staff, and/or LLMs
View Full Description & ApplyYou'll be redirected to the employer's site
