Senior Staff Machine Learning Engineer, Data & Eval
New
A
AirbnbMachine learning
This position is US - Remote Eligible. ... you must live in a state where Airbnb, Inc. has a registered entity.Full-TimeStaff
Salary$244,000 — $305,000 USD
Apply NowOpens the employer's application page
Job Details
- Experience
- 10+ years building, testing, and shipping ML/AI systems end-to-end; including 2+ years of experience with GenAI/LLM systems in production. 5+ years leading large, ambiguous technical initiatives as a senior IC.
- Required Skills
- Machine LearningA/B testingLLMGenerative AI
Requirements
- PhD in Computer Science, Mathematics, Statistics, or a related technical field, or equivalent practical experience.
- 10+ years building, testing, and shipping ML/AI systems end-to-end.
- 2+ years of experience with GenAI/LLM systems in production.
- 5+ years leading large, ambiguous technical initiatives as a senior individual contributor and influencing roadmap and engineering or science direction across teams.
- Deep expertise in evaluation methodology, including offline/online alignment, metric design, human-in-the-loop evaluation, A/B testing, power analysis, and regression testing.
- Hands-on experience with GenAI systems, including orchestration, retrieval, tool calling, and memory.
- Experience building data pipelines and quality systems, including labeling workflows, dataset curation, versioning, monitoring, and governance.
- Solid ML fundamentals and best practices in model selection, training and serving, monitoring, reliability, and model lifecycle management.
- Preferred: experience applying ML/AI to customer support workflows such as agent assist, classification and routing, resolution recommendations, or quality assurance.
- Preferred: experience building evaluation platforms for agent behavior validation, safety and guardrails, and continuous improvement.
Responsibilities
- Define evaluation strategy and success metrics for GenAI systems, aligning offline evaluation with online business and customer experience outcomes.
- Build and scale evaluation frameworks using golden sets, synthetic data, automated regressions, rubric-based grading, and LLM-as-judge where appropriate.
- Apply controls for bias, drift, and reliability in evaluation frameworks.
- Design the data flywheel, including instrumentation, feedback collection, data quality checks, labeling strategy, dataset versioning, and governance.
- Lead cross-functional quality initiatives across product, operations, and engineering.
- Develop and productionize pipelines for dataset creation, model monitoring, evaluation at scale, and continuous pre-deploy and post-deploy testing.
- Drive technical decisions and architecture for evaluation and data infrastructure, balancing speed, rigor, cost, and safety.
View Full Description & ApplyYou'll be redirected to the employer's site
