Lead Engineer, AI Quality
New
C
CircleAI, Software Development
Europe, Middle East, Africa, Asia-Pacific (EMEA, APAC)Full-TimeLead
Salary$170K; $170K • Offers Equity; Annual cash compensation (USD), paid semi-monthly: $170K • Offers Equity
Apply NowOpens the employer's application page
Job Details
- Languages
- English (CEFR Level C2 / ILR Level 5)
- Experience
- 7+ years
- Required Skills
- PythonMachine LearningRuby on RailsCI/CDLLM
Requirements
- 7+ years of experience building and shipping production software.
- Experience with LLM-powered agents that take real actions in a product.
- Experience with complex, tool-using systems involving multiple tools, planning, or orchestration.
- Proficiency in Ruby on Rails or Python.
- Experience building evaluation or observability infrastructure for ML/AI systems.
- Familiarity with evaluation frameworks like Braintrust, LangSmith, or similar.
- Experience designing datasets, annotation workflows, or labeling pipelines for ML/AI evaluation.
- Ability to learn fast and experiment aggressively in a dynamic environment.
- Proficiency in English (CEFR Level C2 / ILR Level 5).
Responsibilities
- Build and own evaluation infrastructure including CI/CD pipelines, scorers, and datasets for AI agents.
- Diagnose failures across agent pipelines including plan creation, tool selection, and execution trajectories.
- Grow datasets via annotation workflows and AI-generated/simulated conversations.
- Run structured experiments across prompts, models, and the agent harness to optimize cost, latency, and performance.
- Lead and grow the AI Quality engineering team while remaining hands-on.
- Partner with AI Core engineering to validate product changes and quality improvements.
View Full Description & ApplyYou'll be redirected to the employer's site
