Member of Technical Staff, Realm Lab
R
Realm LabAI research
Source API remote eligibility restrictions: United StatesFull-Time
SalaryBase salary of $240,000 –$350,000. All employees are eligible for equity compensation, and employees may also receive performance-based bonuses, dependent on role and subject to company policies.
Apply NowOpens the employer's application page
Job Details
- Required Skills
- PythonArtificial IntelligenceMachine Learning
Requirements
- Have a strong technical background in computer science, machine learning, AI, or a related field.
- Bring hands-on experience designing ML evaluations, benchmark suites, research datasets, or experimental frameworks.
- Demonstrate proficiency in Python.
- Have experience building data pipelines, evaluation tools, or ML research infrastructure.
- Understand LLMs, foundation models, post-training techniques, and evaluation methodologies.
- Be able to translate complex real-world workflows into structured tasks, measurable criteria, and research experiments.
- Bring analytical skills in model failure analysis, experimental design, and evaluation quality.
- Be able to work independently and collaborate with researchers, engineers, and domain experts.
- Preferred: Experience with reinforcement learning environments, reward modeling, simulators, or feedback-driven training systems.
- Preferred: Experience evaluating computer use agents, tool use, browser automation, or multi-step agentic workflows.
- Preferred: Familiarity with multimodal reasoning, long-context evaluation, instruction following, or deep research benchmarks.
- Preferred: Experience with human-in-the-loop evaluations, expert-generated datasets, or post-training data.
- Preferred: Contributions to AI research publications, open-source benchmarks, or evaluation frameworks.
Responsibilities
- Own research and evaluation initiatives from task design and dataset development through benchmark validation and performance analysis.
- Design and validate capability evaluations, RL environments, and benchmark suites for computer use, agentic workflows, and emerging model capabilities.
- Develop datasets, task specifications, evaluation rubrics, and methodologies to improve post-training data quality and model assessment.
- Analyze model behavior through failure analysis, error taxonomies, and capability gap identification.
- Collaborate with researchers and domain experts to design experiments, calibrate evaluation quality, and improve model performance.
- Build reproducible evaluation pipelines and feedback loops for iterative model development.
- Contribute to benchmarks, research papers, technical reports, and open research artifacts.
View Full Description & ApplyYou'll be redirected to the employer's site
