QA Automation Engineer (TypeScript)
New
Praga, Bucharest, Sofia, Vilnius, TallinnFull-TimeMiddle
Salary not disclosed
Apply NowOpens the employer's application page
Job Details
- Languages
- English (B2/Intermediate+)
- Experience
- 3+ years
- Required Skills
- CypressTypeScriptAPI testingSeleniumCI/CDLLMPlaywright
Requirements
- Strong experience in QA Automation for AI/LLM-powered SaaS platforms and agentic systems for 3+ years.
- Hands-on experience with automation frameworks such as Playwright, Cypress, Selenium, or similar tools.
- Solid understanding of autonomous and multi-agent workflow validation, including prompt orchestration, tool calling, memory/context handling, RAG systems, and human-in-the-loop processes.
- Experience testing non-deterministic AI behavior and building reliable validation strategies for probabilistic outputs.
- Ability to design and maintain LLM evaluation frameworks focused on reasoning quality, hallucination detection, accuracy, consistency, and response safety.
- Strong experience with API testing, async pipelines, webhooks, queues, and event-driven architecture.
- Experience validating long-running workflows, retry/fallback logic, conversation state persistence, and failure recovery scenarios.
- Strong debugging and troubleshooting skills across frontend, backend, infrastructure, and LLM layers.
- Experience with CI/CD pipelines, automated regression testing, tracing, observability, and log analysis.
- Strong understanding of how testing agentic AI systems differ from traditional QA approaches.
- Level of English – from Intermediate+ and above.
Responsibilities
- Designing, developing, and maintaining automated testing frameworks for AI/LLM-powered SaaS platforms and agentic systems.
- Validating end-to-end autonomous workflows, including prompt orchestration, tool calling, memory and context handling, RAG pipelines, and human-in-the-loop processes.
- Building automated evaluation pipelines to assess reasoning quality, hallucinations, accuracy, consistency, and response safety across non-deterministic AI systems.
- Creating and maintaining regression suites covering long-running workflows, retry and fallback logic, conversation state persistence, and failure recovery scenarios.
- Performing API, async pipeline, webhook, queue, and event-driven system testing across distributed architectures.
- Investigating, reproducing, and debugging issues across frontend, backend, infrastructure, and LLM layers.
- Collaborating closely with AI engineers, backend developers, and product teams to improve quality, reliability, observability, and scalability of AI-driven features.
- Contributing to CI/CD automation, tracing and monitoring solutions, synthetic and golden dataset generation, response scoring, and token/cost analysis for production AI workflows.
View Full Description & ApplyYou'll be redirected to the employer's site
