Lead Product Manager Agent Supply & Quality
New
P
PencilMarketing Technology
EuropeFull-TimeLead
Salary not disclosed
Apply NowOpens the employer's application page
Job Details
- Required Skills
- Product ManagementGenerative AI
Requirements
- Hands-on AI product experience in production.
- Experience shipping LLM or agentic products.
- Understanding of evaluation, reliability, and scaling for high user volume.
- Marketing or creative tech background.
- Experience building evaluation frameworks.
- Partnership and ecosystem thinking with external developers.
- Technical credibility in model selection, prompt architecture, and integration design.
- Rigorous, data-driven approach to product development.
Responsibilities
- Own the By Pencil agent roadmap — quality bar, evaluation frameworks, improvement loops, and the release criteria.
- Define and scale evals — build repeatable, automated evaluation pipelines.
- Drive 3P integrations — own the pipeline of third-party integrations from prioritisation through to launch.
- Build the ecosystem — own the agent builder experience for 1P and 3P developers.
- Own coverage strategy — identify the gaps in agent coverage.
- Work backwards from the customer — write clear customer problem statements.
- Partner with engineering early — provide technical context and understand constraints.
- Instrument everything — define metrics for quality and coverage.
- Prototype don’t explain - build working and actionable prototypes.
View Full Description & ApplyYou'll be redirected to the employer's site
