Staff Data Scientist, Agent Improvement Loop
Y
YaloConversational AI
Location: LATAM, Remote working is available (-+3 hours CT)Full-TimeStaff
Salary not disclosed
Apply NowOpens the employer's application page
Job Details
- Experience
- 8+ years delivering data science solutions with measurable business impact in cross-functional teams.
- Required Skills
- PythonSQLTypeScriptData science
Requirements
- Hold a BS or MS in a technical field such as Computer Science, Statistics, Applied Math, Operations Research or Economics.
- Have 8+ years delivering data science solutions with measurable business impact in cross-functional teams.
- Bring hands-on experience evaluating LLM-based systems or agents, including evaluation design, LLM-as-judge, rubric design and judge calibration.
- Have expert-level Python and SQL skills.
- Be comfortable with TypeScript or ready to work with it.
- Have strong production engineering habits, including version control, code review, testing, CI/CD and shipping tools others use.
- Have production experience with GenAI systems, including prompt and tool design, agent frameworks such as LangGraph, and LLM observability tooling such as LangSmith.
- Be skilled at turning messy trace and conversation data into clear findings for technical and non-technical audiences.
- Bring product and business sense and frame quality work in terms of customer outcomes.
Responsibilities
- Own the evaluation strategy for agents, including pre-release behavioral testing, production monitoring and conversation-scoring judges.
- Analyze agent traces at scale to identify failure patterns, attribute them to skills, prompts or releases, and rank improvement opportunities by impact.
- Turn analysis findings into fixes and new evaluations, ship them, and measure results using before-and-after and causal methods.
- Improve agent skills, harnesses and CLI using trace evidence, working through owning teams’ repositories and review processes.
- Partner with delivery and customer teams to improve customer flows and make the process repeatable.
- Calibrate and validate LLM judges against human review, measure their stability, report limitations and define evidence standards.
- Write production-quality Python and TypeScript evaluation and analysis tools, with tests and CI, for use in a shared codebase.
- Mentor data scientists and engineers on measurement, experimentation and evaluation practices.
- Help shape quarterly and annual planning for agent quality.
View Full Description & ApplyYou'll be redirected to the employer's site
