- Establish the testing framework from zero: unit, integration, end-to-end, and LLM-specific evaluation pipelines
- Define quality standards, test coverage requirements, and documentation practices in partnership with the Lead Engineer
- Audit the existing platform and identify highest-risk surfaces before next major customer deployment
- Define the team structure and hiring roadmap, and begin executing against it
- Recruit, hire, and onboard QA engineers as the team grows, setting clear expectations and standards
- Mentor junior and mid-level QA engineers, building their ability to own test domains independently
- Act as the quality culture carrier across the full engineering team
- Report directly to the Lead Engineer and participate in product planning to ensure quality is designed in
- Design evaluation frameworks for non-deterministic LLM outputs, including prompt regression testing and model drift detection
- Build automated test suites for the agent orchestration layer, including governance agent audit trail integrity
- Validate the Enterprise Knowledge Graph for data accuracy, retrieval quality, and failure modes
- Own end-to-end testing of the file ingestion pipeline across document types
- Validate streaming response handling, latency thresholds, and graceful degradation
- Test multi-model routing logic to confirm cost-optimized task allocation behaves correctly
- Partner with the Full-Stack Engineer to define and test trust-layer UX standards
- Build reusable test playbooks for Forward Deployed Engineers to use in new customer deployments
- Act as the internal advocate for the non-technical enterprise user
PythonExpress.jsSnowflake+6 more