Member of Technical Staff, Pretraining evaluations

Familiarity with base model evaluations and their differences from post-trained models. Strong statistical skills and experience evaluating scientific experiments related to data collection and model performance. Ability to convey statistical information effectively to a broad audience. Extremely strong software engineering skills. Proficiency in programming languages such as Python and ML frameworks (e.g., PyTorch, TensorFlow, JAX). Excellent communication skills. One or more papers at top-tier AI venues (e.g., NeurIPS, ICML, ICLR).

Understand individual evaluation tasks in the base model evaluation suite, including their measurements, strengths, and limitations. Suggest and implement improvements to the base model evaluation suite by adding new tasks or removing redundant ones. Improve statistical understanding of evaluations and enhance the signal-to-noise ratio of the evaluation suite.