Inspect Cyber: A New Standard for Agentic Cyber Evaluations

AI systems are growing more powerful in the cyber domain. This means they can increasingly be used to defend against cyber threats – but can also be exploited to create them. As capabilities advance, there is an escalating need for more rigorous, realistic, and reproducible evaluations.

The current process for building these cyber evaluations, however, is inefficient, with teams often rewriting extensive setup code each time they design a new benchmark. These inefficiencies slow the pace of research, hinder collaboration, and complicate reliability. Today, we’re releasing Inspect Cyber to standardise and streamline the process of creating and running agentic cyber evaluations – tasks that assess AI agents' abilities to autonomously complete cybersecurity objectives across diverse environments.

The existing process for creating and running agentic cyber evaluations is needlessly complex. Every new evaluation or benchmark requires the same repetitive setup, including creating sandboxed infrastructure, connecting agents to models, and developing run scripts. The lack of standardisation causes several critical issues:

We encourage you to share your ideas and contributions through the GitHub repository . If Inspect Cyber doesn’t fit your workflow or if you have ideas for new features, please open an issue or submit a pull request.

Additionally, since Inspect Cyber is built on top of Inspect, it inherits all of Inspect's powerful functionalities, including support for easily running evaluations on diverse models using custom agents, scoring mechanisms, sandbox environments, and policies for human oversight or baselining.

We’ve developed Inspect Cyber to address these inefficiencies. Built on our robust Inspect evaluation platform, Inspect Cyber provides an intuitive, standardised framework for creating and running agentic cyber evaluations. This allows developers to work faster, focus on the most important aspects of their evaluations, and collaborate more effectively. In a nutshell, the open-source Python package:

We have been using Inspect Cyber internally for several months across a spectrum of evaluation scenarios, from foundational Capture the Flag challenges to increasingly sophisticated cyber ranges simulating Active Directory services, enterprise networks, and critical national infrastructure. It has proven remarkably straightforward, robust, and powerful.

However, we appreciate that Inspect Cyber may not yet adequately serve all user workflows. By open sourcing the package, we invite feedback and contributions from the community. Our goal is to continue lowering the barriers to creating agentic cyber evaluations, enabling researchers to more seamlessly bring their evaluation ideas to life.