AI Quality Assurance & Evaluation Specialist
New
G
Gramian ConsultancyIT Consulting
Brazil. Argentina. Colombia. Chile. Peru. UruguayContract
Salary not disclosed
Apply NowOpens the employer's application page
Job Details
- Languages
- English
- Required Skills
- PythonSQLData AnalysisQA
Requirements
- Experience in technical QA, AI evaluation, data analysis, software testing, or a related technical field.
- Comfortable reading and interpreting Python, SQL, shell scripts, structured data, and execution logs.
- Strong analytical skills, including the ability to validate calculations and reconcile conflicting evidence.
- Experience assessing the correctness and completeness of technical or professional deliverables.
- Strong written English and the ability to provide specific, reproducible feedback.
- High attention to detail when reviewing task requirements, grading behavior, and model outputs.
- Ability to independently investigate issues and make evidence-based decisions.
Responsibilities
- Validate task instructions, source materials, reference solutions, and evaluation criteria for consistency and completeness.
- Review AI agent execution traces, tool calls, and generated deliverables to assess whether outcomes are justified.
- Audit grading logic to identify brittle checks, incorrect expected answers, and unsupported rubric criteria.
- Identify cases where valid alternative solutions are unfairly penalized.
- Investigate discrepancies between model performance, task defects, grader errors, and environment or tool failures.
- Independently assess automated QC findings rather than accepting them without verification.
- Document concise, evidence-backed findings and actionable recommendations.
- Clearly communicate uncertainty and distinguish confirmed issues from potential problems.
- Verify that task revisions resolve previously identified defects.
- Collaborate with relevant teams to improve evaluation quality and reliability.
View Full Description & ApplyYou'll be redirected to the employer's site
