Senior AI Product Manager, Evals & Annotations
New
A
ArizeAI observability
United StatesFull-TimeSenior
SalaryThe estimated annual salary for this role is between $200,000-$250,000, plus a competitive equity package.
Apply NowOpens the employer's application page
Job Details
- Experience
- Significant product management experience; flexible on exact years of experience for candidates with exceptional technical depth, product judgment, and ownership.
- Required Skills
- Software ArchitecturePrototyping
Requirements
- Significant product management experience independently owning technically complex, developer-facing products.
- Strong understanding of LLM evaluations and AI development workflows, including automated evals, human feedback, and experimentation.
- Experience with developer tools, AI infrastructure, ML platforms, or adjacent technical products.
- Understanding of backend systems and software architecture.
- Ability to debug issues independently and navigate code when needed.
- Ability to contribute small fixes or pull requests and build prototypes or technical proofs of concept.
- Ability to propose technical implementations and reason through tradeoffs with engineers.
- Strong product and UX instincts, including the ability to define and prototype core product experiences.
- Experience working closely with technical customers.
- Strong ownership, initiative, and ability to navigate ambiguity and make tradeoffs.
- Collaborative approach to working with founders, engineers, and designers.
Responsibilities
- Own the Evals & Annotations product area end-to-end, including eval creation and execution, human review, and annotation workflows.
- Set product strategy and roadmap across customer needs, platform investments, UX improvements, and emerging AI opportunities.
- Develop and prototype evaluation workflows, partnering with Design to deliver polished product experiences.
- Work with Engineering on evaluation execution, backend systems, orchestration, and data infrastructure.
- Talk with customers regularly to understand how they evaluate and improve AI systems and validate product direction.
- Collaborate with founders and cross-functional teams on product direction, design exploration, and prioritization.
- Make prioritization decisions and drive products from problem definition through launch, adoption, and iteration.
- Explore opportunities for evaluations in autonomous and managed agents, including within the Dynatrace ecosystem.
View Full Description & ApplyYou'll be redirected to the employer's site
