- Embed fully within a Scrum team, participating in all ceremonies and maintaining associated artifacts.
- Own the end-to-end test strategy for the AI voice virtual agent across the LLM, speech, and telephony layers.
- Design test scenarios for complex conversational behavior, including multi-turn dialogues, interruptions, and failure recovery.
- Build and maintain an evaluation framework for non-deterministic output, leveraging golden datasets and automated regression detection.
- Adversarially test the agent against hallucinations, guardrail bypasses, and unauthorized data disclosure.
- Prepare and provision SIP devices and endpoints to automate call origination end-to-end.
- Diagnose failures across the entire stack, isolating issues down to the LLM, ASR, TTS, media path, or call routing layer.
- Extend automated coverage and integrate test suites into the CI/CD pipeline.
- Define and track core quality metrics, including task completion, intent accuracy, and response latency.
PythonJenkinsJira+3 more