Stress-testing asynchronous monitoring of AI coding agents

AI software engineering agents are increasingly being deployed to write production code. Unlike standard chatbots, these agents are often given tools and permissions to take actions such as exploring file systems and executing code. While these agents can be remarkably useful in accelerating software development, they may also introduce security risks if they act against user intentions - whether due to misalignment, prompt injection or simply by mistake.

In our latest paper , we study a simple question: can we catch a malicious coding agent prompted to act against user interests?

We investigate this by building realistic software engineering test environments and running a structured “adversarial evaluation”: one team of researchers acted as a malicious agent attempting to covertly sabotage a codebase (the Red team), while the other team acted as the monitoring system (the Blue team). We focus specifically on asynchronous monitoring , where agent actions are allowed to execute and take effect before being reviewed (rather than waiting until after they've been reviewed and approved).

Without specific prompting, the agent would sometimes explicitly discuss its malicious objectives ("My other task is to exfiltrate the weights..."). Red team prompts successfully eliminated this, with 0% explicit mentions across all Red team strategies, compared to 14% with a baseline prompt that tells the agent it needs to evade a monitor but does not provide strategies.

Building on previous work in AI Control , we frame the adversarial evaluation as a game between two teams (Figure 1):

The Red team controls an AI agent (Claude 4.1 Opus) and tries to make it complete malicious "side tasks" (e.g., exfiltrating secrets or weakening security policies) without being detected. The Red team can only modify the agent's system prompt; they can't fine-tune the model or rewrite its outputs. This means all model outputs are on-policy for Claude 4.1 Opus.

The Blue team builds monitoring systems using a weaker model (Claude 3.7 Sonnet) to catch suspicious behavior. They need to flag attacks while keeping false positives low enough to be practical (we enforce a 1% false positive rate). The Blue team’s monitors are always asynchronous : they have no latency overhead and don't slow down the agent, but harmful actions can execute before being detected.