Frontier threats red teaming for AI safety

“Red teaming,” or adversarial testing, is a recognized technique to measure and increase the safety and security of systems. While previous Anthropic research reported methods and results for red teaming using crowdworkers, for some time, AI researchers have noted that AI models could eventually obtain capabilities in areas relevant to national security. For example, researchers have called to measure and monitor these risks , and have written papers with evidence of risks. Anthropic CEO Dario Amodei also highlighted this topic in recent Senate testimony . With that context, we were pleased to advocate for and join in commitments announced at the White House on July 21 that included “internal and external security testing of [our] AI systems” to guard against “some of the most significant sources of AI risks, such as biosecurity and cybersecurity.” However, red teaming in these specialized areas requires intensive investments of time and subject matter expertise.

In this post, we share our approach to “frontier threats red teaming,” high level findings from a project we conducted on biological risks as a test project, lessons learned, and our future plans in this area.

Our goal in this work is to evaluate a baseline of risk, and to create a repeatable way to perform frontier threats red teaming across many topic areas. With respect to biology, while the details of our findings are highly sensitive, we believe it’s important to share our takeaways from this work. In summary, working with experts , we found that models might soon present risks to national security, if unmitigated. However, we also found that there are mitigations to substantially reduce these risks.

We should also prepare for the potential release of models that have not been subject to frontier threats red teaming. We suspect that absent new approaches to mitigation, bad actors could extract harmful biological capabilities with smaller, fine-tuned, or task-specific models adapted from the weights of openly available models if sufficiently capable base models are released.

We are now scaling up this work in order to reliably identify risks and build mitigations. We believe that improving frontier threats red teaming will have immediate benefits and contribute to long-term AI safety . We have been sharing our findings with government, labs, and other stakeholders, and we’d like to see more independent groups doing this work.

Frontier threats red teaming requires investing significant effort to uncover underlying model capabilities. The most important starting point for us has been working with domain experts with decades of experience. Together, we started by defining threat models: what kind of information is dangerous, how that information is combined to create harm, and what degree of accuracy and frequency is required for it to be dangerous. For example, to create harm, it is often necessary to string together many pieces of accurate information, not just generate a single harmful-sounding output.

Following a well-defined research plan, subject matter and LLM experts will need to collectively spend substantial time (i.e. 100+ hours) working closely with models to probe for and understand their true capabilities in a target domain. For example, domain experts may need to learn the best way to interact with or “jailbreak” models.