Senior Product Manager
New
P
PencilAdvertising Technology
EMEAFull-TimeSenior
Salary not disclosed
Apply NowOpens the employer's application page
Job Details
- Required Skills
- Data AnalysisProduct ManagementQuality AssuranceGenerative AI
Requirements
- Proven experience owning a quality, evals, or ML measurement problem as a product with a defined roadmap.
- Experience measuring quality in a subjective domain, defining rubrics, and keeping signal reliable at scale.
- Direct technical experience with golden datasets, rubric design, judge calibration, and inter-rater reliability.
- Ability to derive product decisions from evidenced customer needs rather than solution-first thinking.
- Experience with metrics that drive decision-making outside of your immediate team.
- Experience presenting or defending methodology to clients or buyers.
- Ability to operate in a fast-paced environment with imperfect information and iterative workflows.
- Strong ownership mindset with a proactive approach to identifying and solving problems.
- Design-forward mindset with the ability to ensure products are intuitive and well-polished.
Responsibilities
- Own the charter for Evals & Quality, defining the durable problem, success metrics, and in/out scope.
- Define and drive core quality metrics including first-pass-right, coverage, and regression detection.
- Design and maintain judging methodology, rubrics for creative quality, and calibration processes.
- Implement human QC mechanisms and integrate human judgments into the automated evaluation layer.
- Curate, version, and maintain the golden dataset across various formats and client contexts.
- Build self-service evaluation loops for other teams to utilize when shipping new skills.
- Manage adevals.ai as a product to demonstrate quality to the market.
- Provide evidence layers for commercial activities like client RFP responses and enablement packages.
View Full Description & ApplyYou'll be redirected to the employer's site
