QA Developer, AI Assistant
New
P
PokaIndustrial AI
Location: Québec & Montréal; 100% remote work if you do not live within the radiuses mentioned above.Full-TimeJunior
Salary$500 home office allowance upon hiring
Apply NowOpens the employer's application page
Job Details
- Languages
- An understanding of English is essential.
- Experience
- 2 to 4 years of relevant experience
- Required Skills
- AWSDockerPythonTypeScriptJira
Requirements
- AEC, DEC or BAC in a field related to the position, or equivalent experience.
- 2 to 4 years of relevant experience.
- Background in test automation or as an SDET.
- Comfort with several programming languages, including Python and TypeScript.
- Curiosity about how LLMs behave and why they fail.
- Ability to judge quality when there is no single answer and explain your reasoning.
- Interest in understanding the needs of supervisors on manufacturing floors.
Responsibilities
- Automate QA for the team's features and build tools that let non-developers add evals and analyze results.
- Build and maintain eval suites based on real user questions, expected behavior, and production edge cases.
- Define answer-quality criteria with the team, write scoring rubrics, and use LLM-as-judge where appropriate, calibrated against human-labeled samples.
- Run regression checks in CI when prompts, tools, retrieval, or the Claude model version change, covering answer quality, cost, and speed.
- Review real conversations, group failures into patterns, and work with developers to turn findings into fixes.
- Contribute to adversarial testing, including prompt injection, customer data leakage, and attempts to make the assistant act improperly.
- Conduct timeboxed exploratory testing of features that change what the assistant can decide or do, including checking action safety when failures or retries occur.
- Join feature kickoffs and help define acceptance criteria, including AI-specific criteria.
View Full Description & ApplyYou'll be redirected to the employer's site
