Content Specialist III | AI Evaluation & Prompting
New
J
JobgetherAI evaluation
Based in United StatesContractMiddle
Salary not disclosed
Apply NowOpens the employer's application page
Job Details
- Languages
- Professional fluency in English is required.
- Experience
- 5+ years of professional experience
- Required Skills
- EditingPrompt EngineeringResearchWriting
Requirements
- Have 5+ years of professional experience in writing, editing, journalism, production, linguistics, STEM, coding, policy, or another relevant subject-matter discipline.
- Hold a bachelor’s degree or have equivalent professional experience.
- Bring deep expertise in at least one subject area to evaluate content accurately as a subject-matter expert.
- Demonstrate strong writing, editing, fact-checking, and research capabilities.
- Have experience writing and refining prompts and assessing how prompt changes influence AI behavior.
- Apply detailed rubrics, guidelines, and evaluation criteria consistently across high-volume work.
- Identify subtle differences in AI responses and determine why a response succeeds or fails.
- Verify claims against original or authoritative sources.
- Explain findings, questions, blockers, and recommendations clearly.
- Work independently and escalate important questions, risks, and blockers early.
- Adapt to shifting priorities as AI products and business needs evolve.
- Demonstrate professional fluency in English.
Responsibilities
- Test new AI model versions across topics, use cases, and conversation scenarios.
- Evaluate AI-generated responses against established rubrics, guidelines, and quality standards.
- Document and communicate examples of successful and unsuccessful model behavior.
- Write, test, and refine system prompts to influence model personality, tone, responses, and behavior.
- Assess evaluation rubrics and quality frameworks and identify opportunities for improvement.
- Review human- and agent-based evaluations for accuracy, consistency, and compliance.
- Experiment with AI models and products to investigate capabilities, limitations, and unexpected behaviors.
- Investigate model failures, identify potential causes or patterns, and document findings.
- Fact-check model claims against reliable original sources when required.
- Translate observations and research findings into actionable feedback for product and AI teams.
View Full Description & ApplyYou'll be redirected to the employer's site
