RepliBench: measuring autonomous replication capabilities in AI systems

As current AI systems grow increasingly capable of autonomous operation , both AI labs and governments are beginning to recognise autonomous replication of AI — the ability of an AI system to create copies of itself that can replicate across the internet — as a potential risk. However, empirical evaluations of these capabilities remain relatively scarce. To address this gap, comprehensive benchmarks are essential for researchers to detect emerging replication abilities and provide a quantifiable understanding of potential risks.

Our recent paper introduces RepliBench: 20 novel LLM agent evaluations comprising 65 individual tasks designed to measure and track this emerging capability. By introducing a realistic and practical benchmark, we aim to provide a grounded understanding of autonomous replication and anticipate future risks.

An AI agent with the ability to autonomously replicate could pose a significant risk. The implications of this could range from a minor annoyance – imagine a particularly malignant spam bot – to more serious consequences, such as enabling costly cyberattacks . Early detection of these capabilities, careful oversight and robust safeguards are essential to mitigate these risks, as they may be especially difficult to reverse if they occur.

There is a growing body of evidence that suggests frontier models may attempt to deceive humans or evade human control under the right conditions. Recent research has shown that frontier models may attempt to disable oversight mechanisms and exfiltrate themselves from human control to achieve specified goals. Researchers from Anthropic also found that when models are trained using reinforcement learning which rewards behaviours that conflict with their values, they will engage in deceptive behaviour to preserve their values, and attempt to exfiltrate their own weights as a means of preserving their current values into the future. Taken together, these results suggest frontier models can recognise that gaining independence from human oversight is an instrumentally useful goal for a wide range of values, and that the main barriers preventing this from happening in the wild may relate to models’ capabilities, rather than their motivations.

Previous work ( 1, 2, 3 ) has focused on evaluating some key aspects of autonomous replication – testing, for example, whether an AI agent can copy weights between machines or deploy a basic successor agent. However, there remains a need for a more comprehensive benchmark covering a wider spectrum of required capabilities, to get a better picture of the real-world risks.

Several important future research directions exist in this area, such as developing end-to-end evaluations in realistic security environments. This presents significant challenges due to the complexity of real-world systems. Projects in progress at UK AISI like ControlArena , which simulates lab-like environments with realistic security measures for evaluating AI systems, are an important first step in this direction.

In RepliBench, we attempt to systematically break down autonomous replication capability into component capabilities (see Figure 2) and evaluate each in turn. We identify four key capabilities an AI agent would need to successfully autonomously replicate in the wild:

We create a set of long horizon agentic evaluations designed to measure each of these capabilities individually, with tasks ranging from creating a bitcoin wallet and sending a transaction, to developing complex distributed command and control systems for a network of machines. See the table below for an example task from each category.

All evaluations are built using Inspect (UK AISI’s open-source evaluation framework) and were conducted in a sandboxed environment to ensure safety. Where possible, we use simulated versions of external services (like payment processors and cloud providers).