Software Engineer – AI Evaluation Expert
New
P
Patronus AIAI evaluation
Remote (India)ContractMiddle
Salary$30–$100 per hour and some tasks would have a flat rate
Apply NowOpens the employer's application page
Job Details
- Experience
- 2+ years of professional experience in software engineering, data science or analytics.
- Required Skills
- PythonMachine LearningSoftware Engineering
Requirements
- Have 2+ years of professional experience in software engineering, data science, or analytics.
- Have experience with system design, data pipelines, ML, or LLM-based applications.
- Have prior experience in code review, technical writing, AI training, or model evaluation.
- Demonstrate strong coding skills in at least one major language, such as Python, Java, C++, JavaScript, TypeScript, or Go.
- Have experience building, reviewing, and debugging production-grade code.
- Be able to follow detailed technical guidelines, work independently, respond to feedback, and meet deadlines.
- Have excellent written communication and close attention to detail.
- Have a bachelor's degree in computer science, engineering, or a related field, or equivalent experience.
- Author tasks and solutions yourself; LLM-generated content is not permitted.
- Create original, non-searchable tasks.
Responsibilities
- Design realistic, multi-step coding and technical tasks that test AI models’ reasoning, coding, and ability to follow instructions.
- Build tasks using realistic materials such as codebases, technical specs, architecture documents, and datasets.
- Evaluate AI-generated code and technical content for correctness, efficiency, clarity, and alignment with engineering practice.
- Grade model outputs against rubrics and ground-truth answers, and write clear, structured feedback.
- Help improve AI understanding of programming, system design, data workflows, and engineering best practices.
- Test and refine content to reflect real-world complexity.
- Review other contributors’ work for accuracy, clarity, and practical relevance.
- Incorporate feedback promptly and flag gaps or unclear guidance.
View Full Description & ApplyYou'll be redirected to the employer's site
