Lead Engineer, AI
New
C
CircleAI software
Location: Poland
Secondary Locations: Portugal
Workplace: Remote, Collaborate across time zones; highly async.Full-TimeLead
SalaryAnnual cash compensation (USD), paid semi-monthly: $170K • Offers Equity
Apply NowOpens the employer's application page
Job Details
- Languages
- You are proficient in English (spoken, written, and reading) at a CEFR Level C2 / ILR Level 5.
- Experience
- 7+ years of experience building and shipping production software
- Required Skills
- PythonRuby on RailsCI/CD
Requirements
- Have 7+ years of experience building and shipping production software.
- Bring experience with production LLM-powered agents that take real actions in a product, ideally involving multiple tools, planning or orchestration, and sub-agents.
- Be comfortable with Ruby on Rails and Python, or ready to pick them up quickly; Python proficiency is a strong plus.
- Have experience building evaluation or observability infrastructure for ML/AI systems, such as evaluation pipelines, scorers, dashboards, or CI/CD for evaluations.
- Have experience with evaluation frameworks such as Braintrust, LangSmith, or similar.
- Have experience designing datasets, annotation workflows, or labeling pipelines for ML/AI evaluation.
- Be comfortable running frequent experiments in a dynamic environment and learning new technologies as projects require.
- Be proficient in English (spoken, written, and reading) at CEFR Level C2 or ILR Level 5.
Responsibilities
- Build and own evaluation infrastructure, including CI/CD pipelines, scorers, and datasets for AI agents.
- Trace failures across agent planning, execution, tool selection, and tool trajectories.
- Prototype solutions to quality issues or identify priorities for AI Core engineering.
- Create annotation workflows and AI-generated or simulated conversations to expand evaluation coverage.
- Evaluate new and open-source models against the production baseline and build a framework for model-selection decisions.
- Improve cost and latency through model swaps, caching, and routing by plan complexity.
- Set technical direction and manage day-to-day priorities for the AI Quality engineering team while remaining hands-on.
- Partner with AI Core engineers to assess whether product changes improve AI quality.
View Full Description & ApplyYou'll be redirected to the employer's site
