Senior Research Engineer, AI Evaluation
New
S
SystemAI software
United StatesFull-TimeSenior
SalaryThe base salary range for this role is $180,000 to $240,000, alongside equity
Apply NowOpens the employer's application page
Job Details
- Experience
- 3+ years of experience in software engineering, applied machine learning, or AI research
- Required Skills
- PythonPyTorch
Requirements
- Have 3+ years of experience in software engineering, applied machine learning, or AI research.
- Bring a strong statistical foundation and scientific rigor.
- Have a demonstrated track record designing evaluation harnesses or benchmarking suites for LLMs and complex reasoning systems.
- Have experience with LLM-as-a-judge, reference-based metrics, or human-in-the-loop review.
- Demonstrate advanced proficiency in Python.
- Have experience with PyTorch and Hugging Face.
- Have experience with frontier model APIs and RAG pipelines.
- Have experience with experiment tracking or observability tooling.
- Bring strong written communication skills, such as journal article writing, blog posts, or position papers.
- An advanced degree in Computer Science, Machine Learning, Statistics, or another scientific field is a nice-to-have.
- Peer-reviewed research contributions or presentations at machine learning conferences are a nice-to-have.
- Experience with knowledge graphs is a nice-to-have.
Responsibilities
- Design, build, and maintain automated, reproducible evaluation frameworks to grade LLM reasoning, retrieval accuracy, and structured outputs across CDS applications.
- Create general and domain-specific evaluation benchmarks, synthetic test-case generators, and golden datasets.
- Quantify and mitigate edge cases, hallucinations, model drift, and safety regressions.
- Collaborate with AI Research Scientists, ML Infrastructure Engineers, and Product teams to establish standardized offline and online evaluation loops integrated into CI/CD workflows.
- Contribute to research and development supporting the core platform and product suite.
View Full Description & ApplyYou'll be redirected to the employer's site