AI safety

Lead: Jon Niles-Weed

This area studies the mathematical questions of reliability that a model of this kind raises: what can be guaranteed about its output; how often errors survive checking; and how behavior changes under fine-tuning and distillation. It will study learned verifiers under sustained optimization pressure, including reward exploitation, correlated solver–verifier errors, uncertainty calibration, and independent proof audits. Mathematics is a good setting in which to study the behavior and reliability of large models, since ground truth is often available.

The area will also study the risks of a strong open model for mathematics. Two are specific to mathematics. The first is capability transfer: how much post-training on mathematics improves abilities outside it, including security-relevant ones such as finding software vulnerabilities. The second is the direct use of research-level mathematics against systems whose security rests on mathematical hardness assumptions, as in cryptography. The area will develop evaluations for both, and track them through the stages of the core.

Finally, this area is also responsible, together with the core leads, for the safety of the core itself: the evaluations, audits, and release criteria that govern how the model is trained and released. Since released weights cannot be recalled and safeguards can be removed by fine-tuning, these evaluations will test the model after adversarial fine-tuning and measure its risks relative to open models already available.