Introducing ControlArena: A library for running AI control experiments

Today, we’re launching ControlArena , a library for running AI control experiments.

As AI systems become more powerful and autonomous, ensuring that they remain under human control is one of the central challenges of AI security. If powerful AI systems cannot be reliably expected to follow the intended goals of developers, it will be necessary to develop mechanisms to detect or correct unintended behaviours. AI control is an emerging research agenda which seeks to address this problem. While alignment research focuses on preventing unintended behaviours, control research explores how to maintain oversight and containment even when systems are misaligned.

AI control is gaining traction as a promising research approach to mitigate risks from potentially misaligned AI systems, with many promising and tractable research directions. But it has lacked shared tools to make experimentation easy, consistent, and repeatable. ControlArena aims to change this, by lowering the barrier to entry for researchers across AI safety, cybersecurity, and machine learning, and helping to accelerate innovation in this nascent but critical research area.

If you’re new to the field of AI control, we recommend starting with our introduction to control research . Following on from this you can follow our step-by-step tutorial for running your first control experiment in ControlArena. For more advanced use cases, you can find our full documentation here .

Running AI control experiments has historically been time-consuming and resource-intensive, because each study has required building custom infrastructure from scratch.

The typical control experiment is complex to design. It involves creating a simulated environment (“a setting”) in which an AI agent is tasked with legitimate work (“main tasks”), while also having opportunities to pursue undesirable or malicious objectives (“side tasks”). Researchers then implement and evaluate control protocols , which are mechanisms designed to prevent AI agents from succeeding at these malicious side-tasks. Promising techniques used inside these protocols include deploying another AI model to detect and flag suspicious behaviour (“a monitor”) or requiring human approval for consequential actions.

ControlArena streamlines this process by providing a suite of pre-built settings, ranging from the simple programming problems , to more complex infrastructure codebases . The interface allows researchers to iterate rapidly in simple settings and then graduate to more realistic settings with minimal changes in code. It also provides implementations of common control protocols, and a lightweight interface for running control experiments built on top of our popular Inspect platform .