Lead Engineer, AI Platform

New

CircleAI software

Location: AnywhereFull-TimeLead

SalaryAnnual cash compensation (USD), paid semi-monthly: $170K • Offers Equity

Apply NowOpens the employer's application page

Job Details

Languages: You are proficient in English (spoken, written, and reading) at a CEFR Level C2 / ILR Level 5.
Experience: 7+ years of experience building and shipping production software
Required Skills: PythonRuby on RailsCI/CD

Have 7+ years of experience building and shipping production software.
Have experience with LLM-powered agents that take real actions in a product, ideally involving multiple tools, planning or orchestration, and sub-agents.
Be able to describe a complex tool-using system you shipped and how you evaluated whether it was working.
Be comfortable with Ruby on Rails or Python, or ready to learn them quickly; Python proficiency is a strong plus, especially for data and evaluation work.
Have experience building evaluation or observability infrastructure for ML/AI systems, such as eval pipelines, scorers, dashboards, or CI/CD for evaluations.
Have experience with evaluation frameworks such as Braintrust, LangSmith, or similar.
Have experience designing datasets, annotation workflows, or labeling pipelines for ML/AI evaluation.
Be comfortable running experiments in a dynamic environment, learning new technologies, and working with ambiguity.
Be proficient in spoken, written, and reading English at CEFR Level C2 / ILR Level 5.

Build and own evaluation infrastructure for Circle's AI agents.
Design CI/CD pipelines, scorers, and datasets to evaluate agent performance from individual tool calls to multi-turn conversations.
Trace failures across agent planning, execution, tool selection, and tool trajectories, then prototype solutions or identify priorities for AI Core engineering.
Create annotation workflows and AI-generated or simulated conversations to expand evaluation datasets.
Run structured experiments across prompts, models, and the agent harness; evaluate new and open-source models against the production baseline.
Build a framework for deciding when to shift models and pursue cost and latency improvements through model swaps, caching, and routing by plan complexity.
Set technical direction and manage day-to-day priorities for the AI Quality engineering team while remaining hands-on in the code.
Partner with AI Core engineering to assess whether changes improve AI product quality.

View Full Description & ApplyYou'll be redirected to the employer's site