Automation QA Engineer
New
Must be physically located in Poland or Eastern EuropeFull-TimeMiddle
Salary2,800 - 3,600 EUR per month
Apply NowOpens the employer's application page
Job Details
- Experience
- 3+ years
- Required Skills
- PostgreSQLPythonCypressSeleniumAzure DevOpsLLM
Requirements
- 3+ years of experience in backend test automation, data quality engineering, or Python software development
- Mastery of Python for backend testing, scripting, and data manipulation
- Hands-on experience or deep technical familiarity with LLM validation frameworks (specifically RAGAS or directly equivalent automated validation engines)
- Practical experience configuring automated script execution, regression test triggers, and quality gates inside Azure DevOps
- Functional understanding of PostgreSQL, data profiling techniques, and search indexing concepts
- Must be physically located in Poland or Eastern Europe
Responsibilities
- Build and run automated validation test suites using Python frameworks like RAGAS to continuously score pipeline faithfulness, answer relevance, and context precision
- Integrate automated LLM evaluation scorecards directly into the client's Azure DevOps pipelines to prevent regression in code-generation quality
- Develop automated checks to monitor the performance of documentation ingestion pipelines, chunking consistency, and vector store retrieval accuracy
- Generate programmatic quality scorecards and regression reports to satisfy technical criteria at each project gate approval milestone
- Work closely with senior and mid-level ML implementation engineers to isolate, reproduce, and patch chunking or retrieval flaws exposed by the validation metrics
View Full Description & ApplyYou'll be redirected to the employer's site
