Senior Software Engineer II - Applied AI and Evaluations

SmartsheetSoftware, AI

REMOTE, USAFull-TimeSenior

Salary175000 - 245000 USD per year

Apply NowOpens the employer's application page

Job Details

8+ years of software engineering experience
At least 2 years working directly with LLMs in production
Deep, hands-on experience with prompt engineering and context engineering
Strong working knowledge of RAG architectures: chunking strategies, embedding models, retrieval evaluation, and failure diagnosis
Experience building or extending LLM evaluation frameworks, designing scorers, working with golden datasets
Fluency in agent system design
Strong Python skills
Comfortable working in data-heavy environments (Databricks, Delta tables, or equivalent)
Ability to communicate complex quality findings (written and verbal) to both technical and non-technical stakeholders
Strong cross-functional judgment
A bias for clarity in ambiguous situations
BS or MS in Computer Science, a related field, or equivalent industry experience

Own agent quality end-to-end: diagnosis, improvement, and validation across SmartAssist's orchestrator and subagents
Identify failure modes across quality dimensions factual accuracy, completeness, tone, actionability, and latency and prioritize what to fix
Drive quality improvements through prompt engineering, context engineering, and RAG retrieval tuning
Extend and mature our evaluation framework: scorers, golden datasets, regression gates, and online evaluation for production traffic
Close the feedback loop ensure that every change has a measurable, attributable quality signal
Collaborate with our Agent Architecture lead to distinguish quality problems that require prompt/context solutions from those that require structural fixes
Establish repeatable methodology that scales beyond any single agent or subagent

View Full Description & ApplyYou'll be redirected to the employer's site