Senior Machine Learning Engineer, Agentic Evals and Self-Improvement (Nova)
New
I
IterableAI marketing technology
REMOTE - USFull-TimeSenior
Salary150,000 - 250,000 USD per year
Apply NowOpens the employer's application page
Job Details
- Experience
- 5+ years of hands-on MLE experience putting complex ML systems into production at scale
- Required Skills
- PythonMachine Learning
Requirements
- Have 5+ years of hands-on machine learning engineering experience putting complex ML systems into production at scale.
- Hold a master's or PhD in Computer Science, Machine Learning, AI, or a related field.
- Demonstrate strong Python skills and write readable, modular, maintainable code.
- Have experience with modern ML frameworks.
- Bring deep experience evaluating LLMs or agents, including multi-turn behavior, tool use, and reasoning.
- Have experience building automated evaluation, such as LLM judges, reward models, or benchmark infrastructure.
- Have experience with at least one of prompt or context optimization, post-training, preference optimization, or reinforcement learning.
- Keep up with recent research and apply it through reliable changes that measurably improve customer product experience.
- Communicate effectively and collaborate across teams.
Responsibilities
- Build evaluation systems for models and agents, including benchmarks, LLM judges, simulations, trajectory analysis, and regression tests.
- Create a loop that takes failures from diagnosis through fixes to measured improvement.
- Turn evaluation results into automatic optimization of prompts, context, tools, rubrics, and agent harnesses.
- Build synthetic data pipelines for evaluation and post-training, with attention to quality, privacy, and reproducibility.
- Apply research in LLM-as-judge, reward modeling, test-time search, and post-training to production.
- Mentor engineers on the team and help set Nova's technical direction.
View Full Description & ApplyYou'll be redirected to the employer's site
