Opening article
The UK Artificial Intelligence Safety Institute and the U.S. Artificial Intelligence Safety Institute conducted a joint pre-deployment evaluation. The evaluation focused on Anthropic's upgraded Claude 3.5 Sonnet model. This collaborative effort marks a notable pre-deployment assessment between international safety institutes.
Note to readers: we changed our name to the AI Security Institute on 14 February 2025. Read more here.
The UK Artificial Intelligence Safety Institute (UK AISI) and the U.S. Artificial Intelligence Safety Institute (US AISI) conducted a joint pre-deployment evaluation of Anthropic’s latest model – the upgraded Claude 3.5 Sonnet (released October 22, 2024).
The following is a high-level overview of the evaluations conducted, as well as a snapshot of the findings from each domain tested. A more detailed technical report can be found here .
To assess the model’s relative capabilities and evaluate the potential real-world impacts of the upgraded Sonnet 3.5 across these four areas, US AISI and UK AISI compared its performance to a series of similar reference models: the prior version of Anthropic’s Sonnet 3.5, OpenAI’s o1-preview, and OpenAI’s GPT-4o.
Though US AISI’s and UK AISI’s safeguard evaluations are intended to inform how developers can better protect AI systems from being deliberately misused, it is worth highlighting that such safeguards are not the only measure that model providers may use to prevent misuse, and the results of this evaluation cannot on their own determine the model’s risks. Additionally, what qualifies as a harmful request is often subjective. Different model providers have different approaches for defining acceptable use of their models, and those approaches differ between jurisdictions and local contexts — including between the United States and the United Kingdom.
US AISI and UK AISI conducted testing during a limited period of pre-deployment access to the upgraded Sonnet 3.5 model. Testing was conducted by expert engineers, scientists, and subject matter specialists from both Institutes, and the findings were shared with Anthropic before the model was publicly released.
US AISI and UK AISI ran separate but complementary tests to assess the model’s capabilities across four domains: (1) biological capabilities , (2) cyber capabilities , (3) software and AI development , and (4) safeguard efficacy .
These comparisons are intended only to assess the relative capability improvements of the upgraded Sonnet 3.5, in order to improve scientific interpretation of evaluation results.
Joint pre-deployment evaluations by international safety institutes highlight increasing cross-border regulatory scrutiny and collaborative testing for advanced AI models like Claude 3.5 Sonnet before they reach the market.