Senior Software Engineer, AI
New
CanadaFull-TimeSenior
Salary160,000 - 180,000 CAD per year
Apply NowOpens the employer's application page
Job Details
- Experience
- 5+ years
- Required Skills
- AWSPythonGCPMachine LearningLLM
Requirements
- 5+ years of software engineering experience, with significant exposure to production AI/ML systems.
- Strong hands-on experience with LLM-based systems, including prompt engineering, RAG pipelines, and agent orchestration.
- Proven experience building and operating agentic AI systems in production environments.
- Deep understanding of AI evaluation frameworks and metrics.
- Strong Python engineering skills with a focus on clean, testable, and production-grade code.
- Experience with tools such as LangGraph or similar agent orchestration frameworks.
- Familiarity with LLM observability tools (e.g., LangSmith or equivalents).
- Experience designing and working with vector databases and retrieval systems (e.g., Pinecone or similar).
- Strong understanding of cloud infrastructure (AWS, GCP, or equivalent).
- Solid communication skills.
Responsibilities
- Design and implement end-to-end AI evaluation frameworks, including offline evaluation systems, production tracing, and human-in-the-loop feedback loops.
- Define and operationalize key AI performance metrics such as task success rate, hallucination detection, response quality, and business impact indicators.
- Build and maintain evaluation datasets, automated test harnesses, and regression detection pipelines for AI systems.
- Architect reusable agent systems, including multi-step workflows, LLM DAGs, conversational agents, and orchestration patterns.
- Develop and scale retrieval-augmented generation (RAG) systems, including vector database management and retrieval optimization.
- Integrate observability and monitoring tools for LLM-based systems to improve debugging, reliability, and performance tracking.
- Make informed technical decisions across LLM providers, frameworks, and tooling based on trade-offs in cost, latency, and reliability.
- Lead end-to-end delivery of AI engineering projects, from scoping through execution and deployment.
- Collaborate with engineering leadership to define AI system strategy and evaluation methodologies.
- Elevate engineering standards through code reviews, documentation, mentoring, and technical leadership.
View Full Description & ApplyYou'll be redirected to the employer's site
