How to evaluate control measures for AI agents?

Advances in Large Language Models (LLMs) are raising important questions about how to ensure their behaviour aligns with human goals. A key challenge is the potential for 'misalignment ,' where an LLM's objectives diverge from those of its developers. This poses significant risks, which only increase as LLMs become more autonomous and operate with less human oversight. While current mitigations, which aim to align LLM behaviour during training, are proving useful to make LLMs more helpful and less harmful, they may not always be sufficient. In fact, recent research has shown that AI systems might actively conceal strategies to pursue alternative objectives, a phenomenon known as ‘ scheming ’

As LLMs gain more ability to act independently, the potential for harm from misalignment increases. Therefore, it's crucial to develop robust and scalable alignment strategies. 'AI control ' offers an approach complementary to alignment by focusing on constraining an LLM's actions, even if some misalignment persists after training. Although elements of AI control are used today, questions remain about their scalability as LLMs advance.

Our new paper examines the benefits of AI control and explores how to scale and evaluate these techniques for increasingly capable LLM agents. This blog post summarises our findings and discusses key insights from our work.

The field of AI control focuses on designing ' control protocols ' that constrain AI systems and ensure safety even in the presence of scheming. AI control adopts a conservative stance: it assumes that models might be scheming and designs protections accordingly, rather than assuming models can be made wholly safe through training alone.

A particularly attractive feature of the AI control approach is that it may offer the potential to mitigate alignment risks without fundamental research breakthroughs. Our paper contributes to this discussion by outlining how AI control can help ensure safe deployment of increasingly capable autonomous agents.

As AI systems become more capable of causing harm autonomously, AI developers will need increasingly sophisticated control measures and evaluations to justify that the models can be deployed securely. Our framework provides one possible roadmap for this scaling, proposing five distinct AI control levels (ACLs), ranging from models incapable of competent scheming (ACL-0) to models with scheming capabilities vastly above those of humans (ACL-5).