Senior Research Engineer, AI Evaluation

New
S
SystemAI software
United StatesFull-TimeSenior
SalaryThe base salary range for this role is $180,000 to $240,000, alongside equity
Apply NowOpens the employer's application page

Job Details

Experience
3+ years of experience in software engineering, applied machine learning, or AI research
Required Skills
PythonPyTorch

Requirements

  • Have 3+ years of experience in software engineering, applied machine learning, or AI research.
  • Bring a strong statistical foundation and scientific rigor.
  • Have a demonstrated track record designing evaluation harnesses or benchmarking suites for LLMs and complex reasoning systems.
  • Have experience with LLM-as-a-judge, reference-based metrics, or human-in-the-loop review.
  • Demonstrate advanced proficiency in Python.
  • Have experience with PyTorch and Hugging Face.
  • Have experience with frontier model APIs and RAG pipelines.
  • Have experience with experiment tracking or observability tooling.
  • Bring strong written communication skills, such as journal article writing, blog posts, or position papers.
  • An advanced degree in Computer Science, Machine Learning, Statistics, or another scientific field is a nice-to-have.
  • Peer-reviewed research contributions or presentations at machine learning conferences are a nice-to-have.
  • Experience with knowledge graphs is a nice-to-have.

Responsibilities

  • Design, build, and maintain automated, reproducible evaluation frameworks to grade LLM reasoning, retrieval accuracy, and structured outputs across CDS applications.
  • Create general and domain-specific evaluation benchmarks, synthetic test-case generators, and golden datasets.
  • Quantify and mitigate edge cases, hallucinations, model drift, and safety regressions.
  • Collaborate with AI Research Scientists, ML Infrastructure Engineers, and Product teams to establish standardized offline and online evaluation loops integrated into CI/CD workflows.
  • Contribute to research and development supporting the core platform and product suite.
View Full Description & ApplyYou'll be redirected to the employer's site
The base salary range for this role is $180,000 to $240,000, alongside equity
Apply Now