Lead Engineer, AI Quality
New
C
CircleSaaS Platform
We seek exceptional individuals around the worldFull-TimeLead
Salary$170K
Apply NowOpens the employer's application page
Job Details
- Languages
- English (CEFR Level C2 / ILR Level 5)
- Experience
- 7+ years
- Required Skills
- PythonRuby on RailsCI/CD
Requirements
- 7+ years of experience building and shipping production software.
- Experience with LLM-powered agents that take real actions in a product.
- Proven experience with complex, tool-using systems (multiple tools, planning, orchestration).
- Proficiency in Ruby on Rails or Python.
- Experience building evaluation or observability infrastructure for ML/AI systems (e.g., Braintrust, LangSmith).
- Experience designing datasets, annotation workflows, or labeling pipelines.
- Proficient in English at a CEFR Level C2 / ILR Level 5.
- Ability to thrive in a highly dynamic, fast-paced, and ambiguous environment.
- Comfortable running aggressive experiments and iterative testing cycles.
Responsibilities
- Build and own evaluation infrastructure including CI/CD pipelines, scorers, and datasets for AI agents.
- Diagnose and trace failures across agent pipelines including planning, tool selection, and execution.
- Develop annotation workflows and AI-generated datasets to scale evaluation capabilities.
- Conduct structured experiments across prompts, models, and agent harnesses to optimize for quality, cost, and latency.
- Lead and manage the AI Quality engineering team while maintaining hands-on coding contributions.
- Partner with AI Core engineering to validate improvements and build confidence in AI feature quality.
View Full Description & ApplyYou'll be redirected to the employer's site
