SWE-Bench AI Task Auditor

JobgetherAI systems

Listing location: IndiaContract

Salary60 USD per hour

Apply NowOpens the employer's application page

Job Details

Have demonstrable professional software engineering experience, including navigating complex codebases and developing real-world applications.
Know software engineering principles and be familiar with SWE-Bench-style tasks and evaluation workflows.
Be able to investigate complex technical scenarios systematically using strong analytical and problem-solving skills.
Have experience troubleshooting code, diagnosing test failures, and identifying underlying logic or integration issues.
Be able to evaluate technical work objectively and communicate findings clearly and constructively.
Bring strong attention to detail and a rigorous approach to technical validation.
Be able to work independently and manage project-based assignments effectively.
Have deep expertise in one relevant software engineering specialty; expertise across multiple domains is not required.
Have a secure computer and reliable, high-speed internet connection suitable for remote technical work.

Evaluate software engineering tasks for technical accuracy, realism, solvability, reproducibility, and alignment with SWE-Bench standards.
Review task codebases, integrations, tests, and evaluation criteria to identify technical weaknesses.
Test and troubleshoot complex technical scenarios to determine whether tasks function as intended.
Investigate codebase integration problems, test failures, logic errors, and other implementation issues.
Provide clear, precise, actionable feedback to help task creators correct identified problems.
Apply software engineering judgment to assess whether tasks reflect realistic development scenarios.
Help maintain technical rigor and accuracy across AI training and evaluation workflows.

View Full Description & ApplyYou'll be redirected to the employer's site