Senior Applied AI Engineer, Agent Quality & Evaluations
New
F
FlodeskEmail Marketing
Remote, with in-person hubs in Da Nang (Vietnam), Barcelona (Spain), and Menlo Park (California).Full-TimeSenior
Salary$150,000 - $250,000 base salary, depending on your location and experience. Tier 1 city residents: $165,000 - $250,000. All other US based locations: $150,000 - $235,000.
Apply NowOpens the employer's application page
Job Details
- Required Skills
- PythonTypeScriptPrompt EngineeringLLM
Requirements
- Track record of building or meaningfully improving production LLM or agentic products.
- Strong software engineering skills in Python, TypeScript, or a similar language.
- Ability to build reliable internal tools, evaluation systems, and prototypes.
- Experience with LLM evaluation, observability, and experimental design.
- Strong product sense to translate feedback into testable quality definitions.
- Systems thinking ability to create reusable patterns.
- Clear communication regarding technical tradeoffs like quality, latency, reliability, and cost.
- Experience building content-generation or marketing tools is a bonus.
- Experience with multi-turn agents that use tools is a bonus.
- Experience with Langfuse, Braintrust, or similar platforms is a bonus.
Responsibilities
- Manage member-facing AI behavior across prompt-to-create, agentic editing, and future features like segmentation and analytics.
- Create, test, version, and document prompts and agent instructions with clear improvement tracking.
- Define system use of context, member data, and tools; build and prototype agent loops and structured interfaces.
- Own AI evaluation practice including representative datasets, behavioral scenarios, scoring rubrics, and regression suites.
- Build and operate evaluation harnesses, monitoring, and quality dashboards to measure performance.
- Diagnose failures across prompts, context, orchestration, models, and tools to implement fixes.
- Set quality baselines and release gates for prompt, model, and agent changes.
View Full Description & ApplyYou'll be redirected to the employer's site
