Senior AI QA Test Automation Engineer

New

XMSoftware Testing AI

Warszawa, 1, Warszawa, Country code: PLFull-TimeSenior

Salary not disclosed

Apply NowOpens the employer's application page

Job Details

Languages: English C1
Experience: 6+ years
Required Skills: AWSDockerPythonArtificial IntelligenceQACI/CDGitLabLLMMLOps

BSc/MSc in Computer Science, AI, or a related discipline.
6+ years of hands-on experience in QA/Test Automation.
1+ years of experience applying AI/ML in software testing or QA process improvement.
Strong Python skills with experience in Python/AWS-centric environments.
Hands-on experience with LLM evaluation techniques, including LLM-as-a-judge, RAG, and multi-agent orchestration.
Practical experience with DeepEval or comparable evaluation frameworks and agent-orchestration SDKs.
Knowledge of LLM tooling, vector databases, and MLOps pipelines.
Experience integrating AI tooling into enterprise CI/CD environments like GitLab.
Experience working with containerized cloud-native environments such as Docker and Kubernetes/EKS.
Strong communication skills with the ability to influence technical decisions.

Act as the primary technical enabler for QA, building scalable AI/ML frameworks, libraries, and tooling for the CX Agent Suite.
Design and build AI evaluation pipelines that assess multi-agent responses for accuracy, relevance, tone, safety, and task completion.
Evaluate and prototype agent-orchestration frameworks like Strands Agents to build autonomous QA agents.
Collaborate with QA, Data Science, and Engineering to integrate AI-driven testing into GitLab CI/CD pipelines.
Build resilient automation that adapts to changes in agent behavior and routing logic.
Develop data-driven quality analytics leveraging observability tools like OpenTelemetry, CloudWatch, and LangFuse.
Lead research into emerging AI testing methodologies and mentor engineers through reviews and workshops.

View Full Description & ApplyYou'll be redirected to the employer's site