AI Quality Analyst
New
United StatesFull-TimeMiddle
Salary not disclosed
Apply NowOpens the employer's application page
Job Details
- Experience
- 4+ years of experience in QA, analytics, business process validation, AI evaluation, operations, or similar roles
- Required Skills
- Data AnalysisQuality AssuranceFinancial analysis
Requirements
- 4+ years of experience in QA, analytics, business process validation, AI evaluation, operations, or similar roles.
- Strong writing, analysis, and pattern-recognition skills.
- Experience evaluating outputs against nuanced criteria rather than only binary correctness.
- Ability to work with structured rubrics, scenario libraries, and evidence-based reviews.
- Comfort collaborating across Engineering and business teams.
- Experience with finance, accounting, FP&A, transaction services, or business process design preferred.
- Bachelor's degree preferred.
Responsibilities
- Design and execute structured evaluations for AI-enabled features and workflows.
- Assess outputs for groundedness, instruction adherence, consistency, usefulness, tone, control compliance, and risk.
- Identify hallucinations, unsupported assertions, missing logic, and unsafe recommendations.
- Build and maintain golden datasets, benchmark prompts, comparison sets, and scorecards.
- Develop rubrics that allow quality to be measured consistently across releases and changes.
- Compare performance across prompt versions, workflow revisions, tools, and models.
- Support release decisions with evidence on quality regression or improvement.
- Work closely with Finance SMEs, product managers, and engineers to determine what acceptable looks like in real business contexts.
- Help define human-review thresholds and escalation patterns for higher-risk use cases.
- Analyze reviewer feedback, override patterns, and live quality signals to improve evaluation coverage over time.
View Full Description & ApplyYou'll be redirected to the employer's site
