Research Engineer, Benchmarks
New
H
HUDAI benchmarks
Secondary Locations: Singapore, Remote (Asia), Remote (North America); open to remote candidates who can work hours that 70-80% overlap with either San Francisco or Singapore time zones., 70-80% overlap with either San Francisco or Singapore time zonesFull-Time
Salary100,000 - 230,000 USD per year
Apply NowOpens the employer's application page
Job Details
- Required Skills
- DockerPythonLinux
Requirements
- Demonstrate proficiency in Python, Docker, and Linux environments.
- Have experience working on environments and evaluations.
- Show a strong understanding of what makes benchmarks realistic, reliable, and useful.
- Provide links to published papers or technical blogs on relevant topics, such as public benchmarks and their limitations or model failure modes.
- Be curious and able to understand workflows across different domains.
- Be detail-oriented and able to identify subtle task inconsistencies or edge cases.
- Reason from first principles about task design, scoring, and failure modes.
- Work independently in unstructured problem spaces; early-stage startup experience is a plus.
Responsibilities
- Own the design, implementation, and quality of internal agent benchmarks.
- Work with subject-matter experts to define tasks and create domain-specific benchmarks for realistic workflows.
- Build infrastructure to reliably run models and agents against benchmark tasks.
- Develop metrics and analyses to understand benchmark difficulty, reliability, and failure modes.
- Validate whether benchmark performance correlates with real-world evaluations, customer needs, and lab expectations.
- Write documentation and benchmark reports for technical audiences.
View Full Description & ApplyYou'll be redirected to the employer's site
