Principles for safeguard evaluation

Note to readers: we changed our name to the AI Security Institute on 14 February 2025. Read more here.

At the AI Safety Institute, along with evaluating model capabilities, we’ve been evaluating misuse safeguards —technical interventions implemented by frontier AI developers to prevent users eliciting harmful information or actions from models. We expect safeguards to become increasingly important as AI capabilities advance, and we’re committed to strengthening both our own and others’ ability to evaluate them rigorously. As with many areas of machine learning, we believe that establishing clear problem statements and evaluation frameworks will help drive progress.

Our new paper Principles for Evaluating Misuse Safeguards of Frontier AI Systems proposes a core set recommended best practices to help inform how frontier AI safeguards are measured. To make it easy for frontier AI developers to use these recommendations, we have created a Template for Evaluating Misuse Safeguards of Frontier AI Systems , which draws on these principles to provide a list of concrete and actionable questions to guide effective safeguards evaluation. Our work draws from our experience evaluating the safeguards of a range of frontier AI systems in both pre- and post-deployment tests (e.g. Claude 3.5 Sonnet and our May update ).

We hope these resources will help to drive standardisation in how safeguard evaluations are performed by developers, and how they are presented — internally within developers and to third parties such as collaborators and evaluators. Collaboration is much easier with established frameworks, measurements, and definitions, and these principles are a step in that direction.

We engaged with frontier AI developers and other organisations in the safeguards space to help develop these principles. However, both safeguards and safeguard evaluations are rapidly evolving, and we expect to update these resources as the field advances. We encourage organisations to use our framework and share feedback on how it can be improved so the community moves towards standardised and rigorous safeguards evaluations.

In this post, we outline our proposed 5-step process for safeguards evaluations. This process is designed to be generically useful across threat models, as well as adaptive to changes in the risk landscape.