Quality Analyst II
New
H
HighLevelSaaS, AI Technology
United StatesFull-TimeMiddle
Salary not disclosed
Apply NowOpens the employer's application page
Job Details
- Experience
- 2+ years of experience in Quality Assurance, Customer Support, AI evaluation or a related role
- Required Skills
- SQLData AnalysisRESTful APIsQuality AssuranceCustomer SuccessPrompt EngineeringLLM
Requirements
- 2+ years of experience in Quality Assurance, Customer Support, AI evaluation or a related role.
- Working understanding of LLM behavior, including prompt design and common failure modes.
- General understanding of databases and data flow between systems.
- Comfort reading code and making minor edits to debug prompt behavior and follow integrations.
- Strong written communication skills with the ability to document technical failure modes clearly.
- Ability to follow structured scoring rubrics consistently while suggesting improvements.
- Self-starter mentality with the ability to manage review volume and prioritize tasks with minimal direction.
- Bachelor’s degree in a related field or equivalent practical experience.
- Familiarity with AI development platforms like Claude or Cursor (preferred).
- Familiarity with RAG, vector databases, or knowledge-base architecture (preferred).
- Familiarity with SQL or similar database querying (preferred).
- Experience with APIs, webhooks, or data integration (preferred).
Responsibilities
- Evaluate AI output across the full call lifecycle, including pre-call briefs, in-call guidance, post-call grading, and transcription-driven follow-ups.
- Classify failure modes such as hallucinations, missing context, and tone mismatches using standardized rubrics.
- Refine and iterate on prompts, knowledge base entries, and skills within the ORA platform to resolve issues independently.
- Identify and document systemic trends and patterns in AI performance to inform leadership and improvement backlogs.
- Partner with RevOps Engineering and the Lead Analyst to define quality criteria and validate improvements for new AI agents and orchestrations.
- Maintain up-to-date knowledge on LLM behaviors, prompt patterns, and emerging AI evaluation techniques.
View Full Description & ApplyYou'll be redirected to the employer's site
