- Build the QA Foundation: Establish the testing framework (unit, integration, end-to-end, AI-specific evaluation pipelines) using Playwright and Vitest.
- Define quality standards, test coverage requirements, and documentation practices in partnership with the Lead Engineer.
- Audit the existing platform and identify the highest-risk surfaces before the next client deployment.
- Define the team structure you will need — onshore vs. offshore mix, roles, and a hiring roadmap — and begin executing against it.
- Design evaluation frameworks for non-deterministic LLM outputs — including prompt regression testing, model drift detection, and output quality scoring.
- Build automated test suites for the agent orchestration layer, including governance agent audit trail integrity and human-override behavior.
- Validate the Company Brain (Memgraph + Qdrant) for data accuracy, retrieval quality, and failure modes under real enterprise data conditions.
- Own end-to-end testing of the data ingestion pipelines that connect to client systems through Nango's 700+ connector integration layer.
- Test multi-model routing logic to confirm cost-optimized task allocation behaves correctly across LLM providers via LiteLLM.
- Recruit, hire, and onboard QA engineers as the team grows, setting clear expectations and working standards.
DockerPostgreSQLPython+13 more