Director, AI Automation Engineering
New
I
ImpiricusHealthcare AI
Remote USFull-TimeDirector
Salary185,000 - 210,000 USD per year
Apply NowOpens the employer's application page
Job Details
- Experience
- 5+ years
- Required Skills
- PythonCI/CDGitHub ActionsPrompt EngineeringLLM
Requirements
- 5+ years building, evaluating, or governing production AI/ML systems.
- Direct experience in agentic AI, LLM-based automation, or autonomous system safety.
- Proficiency in Python and standard testing frameworks (e.g., pytest).
- Experience evaluating LLM outputs including RAG eval, prompt regression, and safety benchmarking.
- Experience building CI/CD pipelines with integrated test gates (e.g., GitHub Actions).
- Ability to operate independently as a senior individual contributor.
- Prior experience standing up a QA or automation function from scratch.
- Familiarity with LLM observability and eval tooling like LangSmith or Braintrust is a plus.
- Experience in healthcare, pharma, or regulated environments is a plus.
- Experience with agent testing or bot lifecycle management is a plus.
Responsibilities
- Own infrastructure to continuously verify autonomous agent actions pre- and post-production.
- Manage the lifecycle and testing of agents and workflows.
- Establish baseline quality metrics and coverage standards for AI initiatives.
- Build automated evaluation frameworks for LLM outputs covering accuracy, safety, and bias.
- Implement RAG evaluation, prompt regression testing, and golden-set benchmarking.
- Develop release-gating infrastructure including staged rollouts and rollback authority.
- Integrate testing into GitHub Actions for model and agent releases.
- Document standards and runbooks to enable team self-service for basic testing.
View Full Description & ApplyYou'll be redirected to the employer's site
