Software Engineer: Paid Interview on AI Evaluation Tasks
New
J
JobgetherAI Testing
Based in the United StatesContractMiddle
Salary75 USD per hour
Apply NowOpens the employer's application page
Job Details
- Required Skills
- Full Stack DevelopmentAnalytical SkillsSoftware Engineering
Requirements
- Professional experience as a software engineer, software developer, or closely related technical professional.
- Hands-on experience building, reviewing, testing, or evaluating realistic programming tasks.
- Experience with code review, software testing, automated testing, or technical evaluation frameworks.
- Familiarity with evaluation harnesses or similar environments used to verify programming solutions.
- Experience in one or more relevant areas such as full-stack development, backend engineering, test automation, systems architecture, or related software disciplines.
- Strong understanding of software engineering principles, technical architecture, code quality, and testing methodologies.
- Ability to identify technical flaws and explain their implications clearly and logically.
- Strong analytical and critical-thinking skills, with the ability to assess technical challenges objectively.
- Comfortable discussing complex technical concepts, coding practices, evaluation methodologies, and engineering standards.
- Ability to provide clear, constructive feedback based on practical professional experience.
- Comfortable participating in a remote, structured research interview and sharing detailed technical observations.
Responsibilities
- Review and assess the quality, accuracy, and realism of programming tasks designed to evaluate AI agents.
- Evaluate coding environments and technical evaluation harnesses for correctness, robustness, and suitability for AI testing.
- Examine provided code structures and walk through the underlying logic, identifying potential flaws, inconsistencies, or technical limitations.
- Assess whether coding challenges accurately reflect realistic software engineering scenarios and industry practices.
- Evaluate the difficulty and complexity of programming tasks to determine whether they provide meaningful tests of engineering capabilities.
- Review the verifiability and technical soundness of evaluation criteria and harnesses.
- Provide clear, detailed feedback on potential improvements to task design, evaluation methodology, and technical implementation.
- Discuss technical architecture, testing approaches, and software engineering practices during the research session.
- Share professional perspectives on what makes coding challenges robust, realistic, and technically meaningful.
View Full Description & ApplyYou'll be redirected to the employer's site
