- Build evaluation frameworks for AI agents, including datasets, sandbox environments and metrics for real-world tasks.
- Design experiments to compare models, agent harnesses, tools and execution environments, and identify regressions and trade-offs.
- Turn qualitative feedback about agent behavior into research questions, experiments and measurable outcomes.
- Investigate agent failures and develop ways to improve performance.
- Build and improve agent harnesses, internal tools and integrations, including Aria and other business agents.
- Develop shared knowledge and reusable skills across agents.
- Work with Infrastructure and Security to help agents operate reliably and safely in production.
PythonMachine LearningContent management+1 more