Software Development Engineer in Test - Agentic Automation
New
J
JobgetherCybersecurity
Remote work opportunities across eligible Canadian locationFull-TimeMiddle
SalaryAnnual base salary ranging from CA$120,000 to CA$170,000, plus variable or incentive compensation, equity, and benefits.
Apply NowOpens the employer's application page
Job Details
- Experience
- At least 5 years of experience in software quality, including 3+ years of hands-on SDET or test automation development.
- Required Skills
- Node.jsPythonJavagRPCCI/CDRESTful APIsJSON
Requirements
- Degree in Computer Science or equivalent practical experience.
- Solid understanding of object-oriented programming, data structures, algorithms, and distributed systems.
- At least 5 years of experience in software quality, including 3+ years of hands-on SDET or test automation development.
- Strong programming skills in at least one of Java, Node.js, or Python; Python is preferred for AI evaluation and agent-related work.
- Experience designing, developing, and maintaining test frameworks integrated into CI/CD pipelines such as Jenkins or GitLab CI/CD.
- Experience testing web services and APIs, including REST, gRPC, and JSON.
- Hands-on performance and load testing experience with tools such as Locust or k6, including workload design and interpretation of latency percentiles, throughput, and error rates.
- A verification-first approach to AI-generated output, distinguishing when deterministic assertions are appropriate and when probabilistic evaluation is needed.
- Experience applying AI technologies to improve decision-making, streamline workflows, increase efficiency, and deliver business outcomes.
- Strong analytical, problem-solving, and collaboration skills, including communicating technical findings.
- Nice to have: eval-driven development, release gating, trajectory and tool-use evaluations, and calibration of LLM-as-judge systems against human review.
- Nice to have: prompt engineering, agent context or tool design, performance testing in CI/CD, observability tools, cybersecurity experience, and familiarity with the listed infrastructure and AI technologies.
Responsibilities
- Design, develop, and maintain automated test suites and frameworks for distributed, cloud-based systems.
- Maintain and extend agentic test generation and analysis capabilities within core testing frameworks.
- Build AI-powered workflows and agent skills for test data setup, failure triage, and release validation.
- Maintain evaluation suites, golden datasets, and scoring pipelines for AI agents.
- Implement CI evaluation gates for changes to prompts, models, tools, and orchestration, incorporating production failures into future evaluations.
- Identify and mitigate unreliable AI outputs, flaky or redundant tests, inaccurate failure analysis, and evaluation bias.
- Design and execute performance, load, and stress tests to identify bottlenecks and validate scalability.
- Test integrations across platform systems, services, data, applications, and customer-facing features.
- Review product, system, and agent designs for testability, evaluation coverage, and opportunities for improvement.
- Share technical knowledge and contribute to team-wide quality and evaluation standards.
View Full Description & ApplyYou'll be redirected to the employer's site
