SWE Task Evaluator
M
MercorAI Research
United StatesContractMiddle
Salary70 - 90 USD per hour
Apply NowOpens the employer's application page
Job Details
- Experience
- 3+ years
- Required Skills
- DockerPythonJavaTypeScriptC++GoSoftware Engineering
Requirements
- 3+ years professional software engineering experience.
- Real open-source contribution or maintainer experience (merged PRs, committer/maintainer roles).
- Strong ability to audit reference patches, test runners, and Docker isolation.
- Fluency across common ecosystems (Python and at least one of Java / Go / TypeScript / C++).
- Familiarity with SWE-Bench (Verified) or similar repository benchmarks (Preferred).
- Maintainer history on major Python OSS (Django, Flask, scikit-learn, sympy, pytest, etc.) (Preferred).
- Prior code-review or task-grading experience (Preferred).
Responsibilities
- Evaluate the quality, correctness, and reproducibility of software-engineering benchmark tasks.
- Assess repository-level tasks, reference patches, test harnesses, and grading integrity.
- Provide clear, rubric-based written feedback to improve AI model training.
- Audit reference patches, test runners, and Docker isolation to detect answer leakage and reward hacking.
- Work independently and asynchronously to meet deadlines and enhance AI model performance.
View Full Description & ApplyYou'll be redirected to the employer's site
