Senior AI Development Lead
New
A
AirbnbAI and machine learning
US - Remote Eligible. You must live in a state where Airbnb, Inc. has a registered entity.Full-TimeLead
Salary$157,000 — $171,000 USD
Apply NowOpens the employer's application page
Job Details
- Experience
- 8+ years in product or program management, analytics, consulting or applied AI
- Required Skills
- PythonSQLData AnalysisRESTful APIsPrompt Engineering
Requirements
- Have 8+ years of experience in product or program management, analytics, consulting, or applied AI.
- Have a proven record of taking technical programs from idea to production and owning outcomes such as backlog, cycle time, or launch speed.
- Be strong in SQL and quality statistics, including accuracy, precision, recall, sampling, and reviewer agreement.
- Have shipped an AI-based workflow, agent, or automation that others rely on, and be able to show before-and-after results and explain what changed when it failed.
- Be technically fluent enough to assess engineering estimates and understand how prompts, pipelines, and evaluations fit together.
- Have experience making messy processes measurable and judging success by business outcomes.
- Be able to turn qualitative feedback, including reviewer comments and disagreements, into actionable quality improvements.
- Write clearly and work effectively with sales, operations, product managers, and engineers.
- Be comfortable taking initiative on new problems and trying solutions hands-on.
- Hands-on Python and API experience is a plus.
- Experience with LLM evaluation, experimentation and measurement, or human-in-the-loop work at scale is a plus.
Responsibilities
- Drive supported AI workflows and features from design to launch, defining quality standards and measurement plans with engineering, product, and business partners.
- Prioritize projects by business impact and make trade-offs with managers and business and product leads.
- Map how teams perform work today and determine how AI can handle it and when people should step in.
- Design and build AI workflows, including prompts, pipelines, and tooling, and monitor quality and drift after launch.
- Build evaluation test sets, define accuracy and agreement metrics, and calibrate LLM judges against human reviewers.
- Partner with engineers on architecture and tooling decisions, and assess estimates and scope.
- Turn unclear guidelines into consistent reviewer criteria and work with subject matter experts on labeling guidelines and human-verified labels.
- Run labeling with vendor teams and assess their work using agreement metrics and spot checks.
- Agree on launch-quality criteria with product teams and provide evidence and recommendations for go/no-go decisions.
- Build reusable evaluation and workflow components, document findings, and help other teams adopt them.
View Full Description & ApplyYou'll be redirected to the employer's site