AI Operations Analyst
New
United StatesFull-TimeMiddle
Salary not disclosed
Apply NowOpens the employer's application page
Job Details
- Experience
- 4+ years
- Required Skills
- Quality AssuranceData analytics
Requirements
- 4+ years of experience in production support, AI operations, business process operations, analytics, QA, or a similar role.
- Strong analytical skills and comfort evaluating nuanced workflow or output issues.
- Ability to work across engineering and business teams in ambiguous situations.
- Experience with AI-enabled workflows, operational analytics, business process validation, or support for workflow-heavy systems.
- Strong written communication and documentation discipline.
- Bachelor's degree preferred.
Responsibilities
- Monitor AI-driven workflows in production for abnormal behavior, trust issues, degraded outputs, and recurring exception patterns.
- Review signals related to output quality, fallback use, low-confidence patterns, and workflow anomalies.
- Escalate issues that could affect live user trust, business quality, or finance-sensitive outcomes.
- Support investigation of output-quality incidents, orchestration failures, prompt or workflow regressions, and tool-call problems in production.
- Help distinguish true software defects from model behavior issues, data-quality problems, or process misunderstandings.
- Partner closely with Engineering, QA, and business stakeholders on judgment-heavy incidents.
- Support human-review patterns, fallback routes, and safe-operating responses for AI-enabled features.
- Help define and apply production review practices for higher-risk workflows.
- Document evidence and issue patterns in ways that support repeatability and better release decisions.
- Identify recurring AI operations issues and recommend improvements to supportability, monitoring, and quality feedback loops.
View Full Description & ApplyYou'll be redirected to the employer's site
