Senior QA Automation Engineer
New
I
IDTAI Voice / Telephony
ONLY ACCEPTING CANDIDATES FROM LATAM., Reliable overlap with US Eastern business hoursContractSenior
SalaryCompetitive salary with annual performance review; B2B Employment ($, gross).
Apply NowOpens the employer's application page
Job Details
- Languages
- Advanced spoken English is required.
- Experience
- 5+ years in QA
- Required Skills
- PythonJenkinsJiraWebRTCGitHub ActionsPlaywright
Requirements
- 5+ years in QA, with recent hands-on ownership of a complex, integrated product.
- Practical experience testing SIP/VoIP systems, including call flows, signaling, media handling, and SIP trunking.
- Ability to troubleshoot using SIP traces or packet captures.
- Demonstrated creativity in test design.
- Proficiency in Python, sufficient to independently build and maintain automation frameworks.
- Experience testing non-deterministic or probabilistic systems.
- Strong analytical skills, including the ability to define and track quality metrics.
- Hands-on experience with formal TMS tools and Jira.
- Professional fluency in English (spoken and written).
- Availability for reliable overlap with US Eastern business hours.
Responsibilities
- Embed fully within a Scrum team, participating in all ceremonies and maintaining associated artifacts.
- Own the end-to-end test strategy for the AI voice virtual agent across the LLM, speech, and telephony layers.
- Design test scenarios for complex conversational behavior, including multi-turn dialogues, interruptions, and failure recovery.
- Build and maintain an evaluation framework for non-deterministic output, leveraging golden datasets and automated regression detection.
- Adversarially test the agent against hallucinations, guardrail bypasses, and unauthorized data disclosure.
- Prepare and provision SIP devices and endpoints to automate call origination end-to-end.
- Diagnose failures across the entire stack, isolating issues down to the LLM, ASR, TTS, media path, or call routing layer.
- Extend automated coverage and integrate test suites into the CI/CD pipeline.
- Define and track core quality metrics, including task completion, intent accuracy, and response latency.
View Full Description & ApplyYou'll be redirected to the employer's site
