Senior AI Systems Quality Engineer
New
Remote-first flexibility allowing you to work from anywhere in the United States.Full-TimeSenior
Salary not disclosed
Apply NowOpens the employer's application page
Job Details
- Experience
- 7+ years
- Required Skills
- AWSPythonMLFlowTypeScriptCI/CDDatabricks
Requirements
- 7+ years of software engineering experience, ideally in backend or platform engineering.
- Strong experience designing and implementing automated testing frameworks for complex systems.
- Proficiency in Python and/or TypeScript.
- Hands-on experience with LLM-based systems, agentic workflows, or non-deterministic AI behaviors.
- Deep understanding of CI/CD systems and automated quality gates.
- Experience building scalable evaluation, regression, or validation frameworks for AI systems.
- Familiarity with cloud-native environments, particularly AWS.
- Strong understanding of AI system risks, security, privacy, and governance.
- Ability to define measurable AI quality thresholds and translate them into release criteria.
- Strong collaboration and communication skills.
Responsibilities
- Design and build production-grade automated testing frameworks, evaluation pipelines, and validation systems across the full AI lifecycle.
- Architect and maintain an AI testing platform integrated with tools such as Databricks and MLflow.
- Develop large-scale, scenario-based test suites to validate agentic workflows, including edge cases and failure modes.
- Define and operationalize quality signals for LLM and AI systems and embed them into CI/CD pipelines.
- Validate non-deterministic system behavior and ensure safe degradation patterns.
- Partner with AI, platform, security, and delivery teams to define system quality standards.
- Enable continuous AI validation across deployment pipelines.
View Full Description & ApplyYou'll be redirected to the employer's site
