Applied AI Engineer
New
D
DscoutUX Research Technology
Dscout is proud to support a remote-first workforce and enable employees to work from almost anywhere. At this time, however, we are unable to hire in the following locations: Montana, Hawaii, Alaska, and Washington DC.Full-TimeMiddle
SalaryA strong and competitive compensation package with a built-in bonus and equity program.
Apply NowOpens the employer's application page
Job Details
- Experience
- 2-5 years
- Required Skills
- PythonMachine LearningData modelingPrompt EngineeringLLM
Requirements
- 2-5 years of software engineering experience, with hands-on experience building or operating LLM-powered features or agents in production
- Fluency with prompting and context engineering as an engineering discipline
- Experience building or maintaining evaluation harnesses for AI systems, such as offline eval sets or LLM-as-judge scoring
- Comfort reasoning about agent behavior across a distribution of production traffic
- Experience running experiments like A/B testing and staged rollouts
- Track record of shipping features that real users depended on
- High-agency mindset to investigate ambiguous underperformance problems
- Comfort using AI coding tools such as Cursor, Claude Code, or Copilot
Responsibilities
- Own the production improvement loop across agent behavior, customer and operator feedback, evaluation, experimentation, and verified business outcomes
- Instrument agent workflows so model interactions, tool use, decisions, failures, human edits, and downstream outcomes can be understood in context
- Define meaningful quality standards, representative evaluation datasets, regression coverage, and production monitoring
- Investigate why agents underperform across context, knowledge, instructions, tools, routing, guardrails, or workflow design
- Design and ship targeted behavior improvements, including changes to prompting, context construction, decision logic, tool use, and human-review paths
- Build backend services, APIs, data models, and feedback pipelines that make agent behavior observable, steerable, and reproducible
- Run controlled experiments, production replays, or staged rollouts to measure whether changes improve quality and downstream business results
View Full Description & ApplyYou'll be redirected to the employer's site
