Staff Data Scientist, Agent Improvement Loop
J
JobgetherConversational AI
Based in Brazil, Working hours offer approximately three hours of flexibility around Central Time.Full-TimeStaff
Salary not disclosed
Apply NowOpens the employer's application page
Job Details
- Experience
- At least 8 years of experience delivering data science solutions with measurable business impact in cross-functional environments.
- Required Skills
- PythonSQLTypeScriptData sciencePrompt Engineering
Requirements
- Bachelor's or Master's degree in Computer Science, Statistics, Applied Mathematics, Operations Research, Economics, or a related technical discipline.
- At least 8 years of experience delivering data science solutions with measurable business impact in cross-functional environments.
- Hands-on experience evaluating LLM-based systems or AI agents, including evaluation design, LLM-as-a-judge methodologies, evaluation rubrics, and judge calibration.
- Advanced proficiency in Python and SQL; working knowledge of TypeScript or willingness to develop expertise in it.
- Strong software engineering practices, including version control, code reviews, automated testing, and CI/CD.
- Experience building and deploying tools for shared production environments.
- Practical experience with production GenAI systems, prompt engineering, tool design, agent frameworks such as LangGraph, and LLM observability platforms such as LangSmith.
- Ability to turn complex, unstructured trace and conversational data into actionable insights and communicate findings to technical and non-technical stakeholders.
- Product and business acumen to connect technical quality improvements to customer experience and measurable business outcomes.
- Experience with conversational AI, customer-facing agents, or tool-using AI systems is an advantage.
- Knowledge of NLP, clustering, and methods for discovering and categorizing recurring system failures is a plus.
- Familiarity with causal inference and experimentation, including A/B testing, difference-in-differences, and pre/post analysis.
- Exposure to MLOps or LLMOps platforms such as Vertex AI, Amazon SageMaker, or MLflow, and to big data tools, is desirable.
- Understanding of AI safety, responsible AI principles, and reliable AI development practices is an advantage.
- Mentoring experience and ability to support other data scientists' and engineers' professional development are valued.
Responsibilities
- Own and develop evaluation strategies for AI agents, including pre-release behavioral testing, production monitoring, and automated evaluation systems.
- Analyze agent traces and conversations at scale to identify recurring failure patterns, determine root causes, and prioritize improvements by business impact.
- Translate analytical findings into fixes, implement improvements, and measure effectiveness through before-and-after comparisons and causal analysis.
- Improve agent skills, orchestration frameworks, supporting infrastructure, and developer tools using production-trace evidence.
- Partner with Delivery and Customer Success to improve customer conversational workflows and establish repeatable performance-maintenance processes.
- Design, calibrate, and validate LLM-as-a-judge systems against human assessments, measuring consistency, reliability, limitations, and effectiveness.
- Develop and maintain production-quality Python and TypeScript evaluation and analytical tools, with testing, documentation, version control, and CI/CD integration.
- Mentor data scientists and engineers in measurement, experimentation, evaluation, and data-driven decision-making.
- Contribute to quarterly and annual strategic planning for AI-agent quality.
View Full Description & ApplyYou'll be redirected to the employer's site
