Senior QA Engineer — AI Systems & Platform
New
P
Peach PilotArtificial Intelligence
Fully Remote US Based, EDTContractSenior
Salary not disclosed
Apply NowOpens the employer's application page
Job Details
- Experience
- 7+ years of QA engineering experience, with at least 3 years in a senior or lead capacity
- Required Skills
- PythonAPI testingCI/CDPrompt EngineeringPlaywright
Requirements
- 7+ years of QA engineering experience.
- 3+ years in a senior or lead capacity.
- Proven experience testing AI/LLM-powered applications.
- Proficiency in Python for test automation.
- Hands-on experience building automation frameworks from scratch with Playwright and Vitest.
- Experience testing complex API chains, async/streaming responses, and multi-service workflows.
- Understanding of prompt sensitivity and output variance.
- Ability to work full-time contract hours with ~5 hours/day EDT overlap.
- Experience with CI/CD pipelines (GitHub Actions).
Responsibilities
- Establish testing framework (unit, integration, E2E) using Playwright and Vitest.
- Define quality standards and test coverage requirements.
- Design evaluation frameworks for LLM outputs, including prompt regression and model drift detection.
- Build automated test suites for agent orchestration and audit-trail integrity.
- Validate knowledge graph and data accuracy for enterprise data.
- Test data ingestion pipelines connected to client systems via Nango.
- Verify multi-model routing logic and streaming response handling.
- Conduct risk assessment and audit platform before client deployments.
View Full Description & ApplyYou'll be redirected to the employer's site
