Technical AI Evaluation Analyst
New
G
Gramian ConsultancyIT Professional Services
Argentina. Brazil. Mexico. Uruguay. Peru. Chile, 8 hours of mandatory PST overlapContractMiddle
Salary not disclosed
Apply NowOpens the employer's application page
Job Details
- Languages
- Strong written English
- Required Skills
- PythonSQLAnalytical SkillsQuality Assurance
Requirements
- Comfortable reading and interpreting Python, SQL, shell scripts, structured data, and execution logs.
- Experience reviewing technical workflows, software behavior, data outputs, or evaluation logic.
- Strong analytical skills, including the ability to verify calculations and reconcile conflicting evidence.
- Demonstrated ability to assess the correctness and completeness of technical deliverables.
- Strong written English with experience providing clear, specific, and reproducible feedback.
- High attention to detail when identifying inconsistencies, missing information, and evaluation defects.
Responsibilities
- Validate task quality by checking instructions, source materials, reference solutions, and evaluation criteria for consistency and completeness.
- Review AI agent execution traces, tool calls, and deliverables to determine whether outcomes are justified.
- Audit grading logic to identify brittle checks, incorrect expected answers, unsupported rubric criteria, and unfair penalties for valid alternative solutions.
- Investigate discrepancies between model performance, grader results, and expected outcomes.
- Distinguish genuine model limitations from task defects, grader errors, and environment or tool failures.
- Independently assess automated QC findings rather than accepting them without verification.
- Document concise, evidence-backed findings and provide actionable, reproducible feedback.
- Flag uncertainty and verify that implemented revisions resolve previously identified issues.
View Full Description & ApplyYou'll be redirected to the employer's site
