Safety cases at AISI

Note to readers: we changed our name to the AI Security Institute on 14 February 2025. Read more here.

The benefits and risks posed by frontier AI models are changing over time as performance increases, requiring corresponding changes to how we mitigate those risks and the evidence required to trust those mitigations. Existing models lack the capabilities required to pose certain severe risks such as irreversible loss of human control, so arguments based on capabilities evaluations suffice. If future models are much more capable then different types of arguments and evidence may be required to understand whether such risks are present.

AI is a fast-moving field with different developers taking different approaches to the safety of frontier AI models. One way to accommodate for this is for AI developers to build safety cases tailored to their specific technical and deployment contexts. The UK Ministry of Defence’s Defence Standard 00-56 defines a safety case as:

For example, a safety case for AI control might use a red team to attempt to break a control system used to contain an AI model. This involves many choices specifying the deployment environment of the model, the types of control mechanisms used, and the limitations imposed on the red team such as whether the red team can iteratively jailbreak the model. We believe it will be important in future safety case work to specify these parameters, to make analyses and countercases more concrete.

Two frontier AI developers have highlighted safety cases as part of their approaches to high-capability models: Anthropic includes “affirmative cases” in their sketch of ASL-4 policies , and Google DeepMind’s Deployment Mitigation Level 2 is “safety case with red team validation”. We view this is a positive development and believe that safety cases are an important (but not the only) part of a developer's overall approach to AI safety.

We expect safety cases for AI systems to combine a variety of types of evidence, including empirical, conceptual, and mathematical arguments for why methods work, negative evidence such as the failure of a well-incentivised red team to break safety methods, and sociotechnical evidence about the deployment context, potential harms, and organisation within the AI company.

However, since our understanding of AI safety is nascent, it is not yet possible to build full safety cases that scale to risks posed by models significantly more advanced than those of today, such as risks due to loss of control or high levels of autonomy. Experience with safety cases for less capable models is helpful but insufficient, as many details will be different. Despite this, we believe we can significantly grow our understanding of what safety cases might look like by building safety case sketches that detail the arguments and evidence we expect for specific safety methods. These sketches could form indicative building blocks for full safety cases in the future.