Applied AI Software Engineer
New
C
Canvas MedicalHealth Tech
San Francisco, CA / RemoteFull-TimeSenior
Salary$300,000 - $400,000 a year
Apply NowOpens the employer's application page
Job Details
- Experience
- 5+ years
- Required Skills
- PythonSQLMachine LearningPrompt EngineeringLLM
Requirements
- 5+ years of experience in applied machine learning or AI engineering, with a focus on evaluation and benchmarking.
- Proficiency with foundation model APIs (e.g., OpenAI, Claude, Gemini).
- Experience orchestrating complex agent behaviors via prompts or tools.
- Experience designing and running high-throughput evaluation pipelines, including human-in-the-loop or expert-labeled benchmarks.
- Superlative Python engineering skills.
- Familiarity with experiment management tools and data engineering toolsets.
- Familiarity with SQL and database management.
- Familiarity with clinical or healthcare data is a strong plus.
- Experience with reinforcement fine-tuning, model monitoring, or RLHF is a plus.
Responsibilities
- Design and execute large-scale evaluation plans for LLM-based agents performing clinical documentation, scheduling, billing, and communication tasks.
- Build end-to-end test harnesses that validate model behavior under different configurations.
- Partner with clinicians to define accurate expected outcomes for performance comparisons in clinical domains.
- Run and replicate experiments across multiple models and parameters to determine optimal configurations.
- Deploy and maintain ongoing sampling for post-deployment governance of agent fleets.
- Analyze results and summarize tradeoffs for product and engineering stakeholders.
- Take ownership over internal evaluation tooling and infrastructure to ensure reproducibility.
View Full Description & ApplyYou'll be redirected to the employer's site
