Senior Data Scientist, AI Evaluation
New
A
AlpacaBrokerage infrastructure
Remote - AmericasFull-TimeSenior
Salary not disclosed
Apply NowOpens the employer's application page
Job Details
- Experience
- 6–10 years in quantitative data science or ML, with focused experience in measurement or evaluation.
- Required Skills
- PythonSQLMachine Learning
Requirements
- Bring 6–10 years in quantitative data science or ML, with focused experience in measurement or evaluation.
- Demonstrate quantitative measurement rigor, such as LLM/model evaluation, metric validation, or experimentation.
- Have a strong statistical and machine learning foundation.
- Treat evaluations as experiments, including sample sizing, confidence intervals, significance, and handling non-determinism.
- Validate automated graders against human ground truth.
- Be proficient in Python and SQL.
- Have experience evaluating models in production environments.
- Use strong judgment to define quality metrics and ground truth for ambiguous outputs.
- Communicate and collaborate across technical teams and leadership.
- A quantitative degree is a plus; equivalent industry experience is welcome.
Responsibilities
- Define ground truth, metrics, and scoring methods for models and agents.
- Build repeatable evaluation loops to track quality over time and catch regressions before release.
- Work with engineering and analytics engineering to operationalize evaluation harnesses.
- Translate evaluation results into actionable recommendations for system improvements.
- Establish evaluation guidelines, documentation, and review practices.
- Foster evaluation best practices and measurable AI quality across the team.
View Full Description & ApplyYou'll be redirected to the employer's site
