Staff AI Product Analyst
New
USFull-TimeStaff
Salary not disclosed
Apply NowOpens the employer's application page
Job Details
- Experience
- 10+ years
- Required Skills
- PythonMachine LearningProduct AnalyticsData scienceLLM
Requirements
- 10+ years of experience in product analytics, data science, applied analytics, or technical product roles in high-scale environments.
- 1+ year of hands-on experience working with AI/LLM systems in applied production or evaluation contexts.
- Strong technical proficiency in Python and experience with data analysis, ML concepts, and analytics tooling.
- Proven experience designing AI evaluation systems, including structured evals, benchmarking datasets, or production monitoring frameworks.
- Strong understanding of data governance, experimentation design, and causal analysis methodologies.
- Ability to translate complex AI and data insights into clear, actionable recommendations for non-technical stakeholders.
- Experience working in cross-functional environments involving product, engineering, and data science teams.
- Strong systems-thinking mindset with the ability to reason about end-to-end data and AI workflows.
- Comfortable operating in ambiguous, fast-evolving AI product environments with iterative development cycles.
Responsibilities
- Define AI-native product analytics strategies, including north-star and guardrail metrics for key AI-powered user journeys, focusing on adoption, quality, and business impact.
- Design and maintain evaluation frameworks for AI systems, including golden datasets, scoring rubrics, regression tests, and structured model/agent assessments.
- Partner with data science and product teams to support experimentation, causal inference, and interpretation of AI product performance.
- Build systems-level understanding of how agents interact with data sources, APIs, tools, and metrics to ensure analytics requirements are clear and actionable.
- Develop taxonomy frameworks to classify AI failure modes such as hallucinations, tool misuse, retrieval gaps, and edge-case behaviors.
- Monitor live AI product usage to track success rates, abandonment patterns, fallback behaviors, and human override signals.
- Own analytics enablement through playbooks, documentation, and frameworks that define how AI systems should be evaluated and interpreted.
- Collaborate with engineering and data teams to ensure reliable data pipelines, consistent definitions, and trustworthy metric systems.
- Produce recurring insights, dashboards, and executive readouts that guide roadmap prioritization and reduce high-severity AI system issues.
- Support cross-functional alignment between technical AI capabilities and business outcomes through clear narrative and data storytelling.
View Full Description & ApplyYou'll be redirected to the employer's site
