Member of Technical Staff, Pretraining evaluations

Familiarity with base model evaluations. Strong statistical skills and experience evaluating scientific experiments. Ability to convey statistical information effectively. Extremely strong software engineering skills. Proficiency in programming languages such as Python and ML frameworks (e.g., PyTorch, TensorFlow, JAX). Excellent communication skills. One or more papers at top-tier venues (e.g., NeurIPS, ICML, ICLR).

Understand individual evaluation tasks in the base model evaluation suite. Suggest and implement improvements to the base model evaluation suite. Improve statistical understanding and signal-to-noise ratio of the evaluation suite.