Lead Product Manager, Agent Runtime & Evals

New

JobgetherAI platform

Based in CanadaFull-TimeLead

SalaryUS salary range of USD $200,000–$230,000+; Canadian compensation details can be confirmed with the recruiter.

Apply NowOpens the employer's application page

Job Details

Experience: 7+ years of product management experience in B2B or enterprise SaaS environments, including at least 3 years working in AI-focused product roles.
Required Skills: Product ManagementRegression testingLLM

Have 7+ years of product management experience in B2B or enterprise SaaS environments.
Have at least 3 years of experience in AI-focused product roles.
Have owned complex products end to end, from vision and strategy through launch and continuous improvement.
Have experience building and scaling AI agents or automation solutions that replace manual business processes.
Understand AI evaluation methods, including golden datasets, data labeling, automated judges, regression testing, performance monitoring, and human-in-the-loop evaluation.
Be able to work with engineering teams on agent-to-workflow contracts, structured outputs, deterministic execution, agent memory, and model orchestration.
Understand how non-technical users build, configure, deploy, and supervise automated systems.
Have hands-on fluency with modern AI tools and large language models, including Claude and Cursor, and be able to prototype solutions independently.
Be able to make informed decisions about AI models, evaluation frameworks, integrations, access controls, and deployment strategies.
Have strong written and verbal communication skills to explain product decisions to engineers, designers, executives, and customers.
Demonstrate leadership, cross-team influence, product direction-setting, and mentoring or management of product managers.
Bring product judgment, analytical thinking, customer empathy, and comfort navigating ambiguity.

Own the vision, product strategy, and roadmap for the agent runtime platform.
Lead agent lifecycle capabilities, including version control, deployment, runtime management, observability, and continuous improvement.
Define evaluation and model governance frameworks, including datasets, labeling, automated evaluation, regression testing, and quality gates.
Establish mechanisms to evaluate agents and measure quality, reliability, and performance.
Shape agent memory, learning, and self-improvement capabilities.
Lead development of a model-agnostic platform, including model routing, cost attribution, and pricing strategies.
Define secure integration strategies for internal systems, business data, and third-party applications.
Establish agent identity and execution models, including permissions, governance, and auditability.
Develop monitoring and oversight experiences with dashboards, histories, traces, metrics, logs, audit trails, and emergency controls.
Partner with workflow orchestration leadership and engineering and design teams on standards and configuration tools; monitor emerging AI technologies and mentor product colleagues.

View Full Description & ApplyYou'll be redirected to the employer's site