Senior AI Product Manager, Evals & Annotations

New

ArizeAI observability

Location: United States (Remote)Full-TimeSenior

SalaryThe estimated annual salary for this role is between $200,000-$250,000, plus a competitive equity package.

Apply NowOpens the employer's application page

Job Details

Experience: Significant product management experience; flexible on exact years of experience for candidates with exceptional technical depth, product judgment, and ownership.
Required Skills: Product ManagementSoftware ArchitecturePrototyping

Bring significant product management experience independently owning technically complex, developer-facing products; exact years are flexible for candidates with exceptional technical depth, product judgment, and ownership.
Understand LLM evaluations and AI development workflows, including automated evals, human feedback, experimentation, and how developers iterate on AI applications and agents.
Have experience with developer tools, AI infrastructure, ML platforms, or adjacent technical products.
Understand backend systems and software architecture.
Be able to debug issues independently and navigate code when needed.
Be able to contribute small fixes or pull requests, build prototypes and technical proofs of concept, and reason about technical implementations and tradeoffs.
Show strong product and UX instincts, including the ability to define and prototype core product experiences.
Demonstrate ownership, initiative, and a bias toward action in ambiguous situations.
Work collaboratively with founders, engineers, and designers, and engage in constructive debate.
Have experience working closely with technical customers and strong communication skills.
Bonus: Experience with LLM evaluation platforms, annotation systems, human-in-the-loop workflows, or agent orchestration.
Bonus: Backend or platform engineering experience, distributed systems, workflow orchestration, data infrastructure, or hands-on programming across backend and frontend technologies.

Own the Evals & Annotations product area end-to-end, including eval creation and execution, human review, annotation workflows, and the evaluation lifecycle.
Set product strategy and roadmap, balancing customer needs, platform investments, UX improvements, and opportunities in AI and agent development.
Develop and prototype evaluation workflows, then partner with Design to deliver intuitive, polished experiences.
Shape evaluation capabilities for autonomous and managed agents, including opportunities in the Dynatrace ecosystem.
Work with Engineering on evaluation execution, backend systems, orchestration, and data infrastructure, contributing to implementation discussions and tradeoffs.
Speak with customers multiple times each week to understand how they evaluate and improve AI systems, uncover problems, and validate product direction.
Collaborate with cofounders and cross-functional teams on product direction, design exploration, and prioritization.
Prioritize competing demands and drive products from problem definition through launch, adoption, and iteration.

View Full Description & ApplyYou'll be redirected to the employer's site