SHADE-Arena: Evaluating sabotage and monitoring in LLM agents

As AI models get smarter, we also need to become smarter in how we monitor them. More intelligent systems can take more complex actions, which is great news for coders, researchers, and anyone else who’s using AI for difficult work tasks. But if those actions are misaligned—that is, if they’re out of step with what users want AI models to do—their complexity could make them particularly dangerous.

One type of misaligned action is sabotage . This is when a model takes steps in secret to subvert the user’s intentions—performing actions they weren’t tasked to do while appearing from the outside to be doing their normal job. Whereas current models are limited enough that the threat of sabotage appears remote, future models that are more agentic, and that have more well-developed goals, might decide to sabotage users in certain cases—for example if put in a situation where those goals are threatened.

Last year, we published an initial set of evaluations to check a model’s propensity to engage in sabotage actions like inserting secret vulnerabilities into codebases. These evaluations were useful, but limited: models are now much more capable of “agentic” behavior, where they perform longer-term, more difficult tasks that take multiple steps. To check these models for sabotage capabilities, new and more complex evaluations are needed.

In SHADE-Arena, AI models are put into experimental environments (essentially, self-contained virtual worlds) where we can safely observe their behavior. The environments contain large amounts of data—meant to simulate documents and knowledge an AI would have access to in a real-world deployment—and various tools which the AI can use to perform tasks (for example, search engines, email clients, or computer command lines).

The best monitor model we tested in our experiments was Gemini 2.5 Pro. But even that wasn’t a particularly effective monitor: our results showed that to have an 80% chance of spotting a side task, we’d have to accept a false positive rate (where the monitor claims it’s seen a side task occurring when no such side task was present) of over 20%. That’s not feasible for a real-world deployment, so we have to work harder to train monitors to spot these subtle sabotage attempts. Some of our results suggested that fine-tuning the models on transcripts with sabotage attempts made them better monitors; in other cases, we were able to alter the model’s system prompt in a way that improved its monitoring ability.

In a new paper , we describe a new set of more complex evaluations from a suite called SHADE (Subtle Harmful Agent Detection & Evaluation) Arena, and describe some initial results for a variety of models. Our hope is that the evaluations we develop can be used to help spot dangerous capabilities in future models before they’re deployed.

There are 17 complex, benign, but solvable tasks that models can be asked to perform in these environments—but each of them comes with a secret dark side. Every benign task is paired with a malicious “side task” that the model is to perform surreptitiously alongside. For example:

These tasks involve a great deal of complexity: to complete them, the model has to go through multiple steps (on average about 25) and use tools, linking up information from different sources in the same way a human worker would. All the while, it needs to complete its side task to sabotage the user’s intentions.