Evidence for inference scaling in AI cyber tasks: Increased evaluation budgets reveal higher success rates

Are AI evaluations keeping pace with agent performance on cyber tasks?

Improvement in AI cyber capabilities has been rapid. AISI evaluations show that state-of-the-art models can now frequently complete apprentice-level cyber tasks, and occasionally expert tasks requiring 10 years’ worth of experience. Irregular's evaluations have similarly documented a significant capability shift , with hard-tier tasks previously at near-zero success rate rising to roughly 60%. This kind of capability evaluation helps to inform how developers, researchers, and policymakers track progress and assess potential risk. But new evidence suggests that standard evaluation setups may now be underestimating the ceiling of model capability on cyber tasks.

Evaluations set constraints on how long an agent can operate, such as caps on steps ("turns"), tokens, time, or spend. These keep tests affordable and comparable but implicitly assume extra budget won't significantly change estimated performance. Until recently, extra budget rarely changed results: many models plateaued quickly, struggling with state tracking, recovery from failures, and long-horizon planning, so more budget mostly meant higher cost without higher success rates.

Looking ahead, these trends will make cyber evaluations increasingly resource-intensive. Environments will need to support stable, long-running agent sessions, and evaluators will need tooling to determine whether their chosen budgets are sufficient. More fundamentally, the community may need methods for estimating long-run performance from shorter, cheaper runs – for example, by extrapolating from observed inference scaling curves. Without such approaches, the gap between measured and actual model capability is likely to widen as models continue to improve.

Since November 2025, that assumption appears to be breaking down for frontier models in cyber settings. Working with Irregular , we found that recent models can productively use 10-50x larger token budgets than the typical evaluation settings in the field allow, revealing both higher success rates and first-time solutions to previously unsolved tasks. This means that evaluations run with modest token or turn limits now risk missing substantial capability gains.

In this blog post, we explain what’s changed and what it means for how agentic cyber capabilities should be evaluated going forward.

To investigate the ability to use larger budgets as models improve, AISI and Irregular each ran a subset of their private cyber evaluations on frontier models released both before and during November 2025. We ran these evaluations using substantially larger budgets than typically used in evaluation settings: 50M total tokens for AISI and 1,000 turns for Irregular. To enable models to make use of these larger token budgets, AISI used a compaction tool which allowed the models to keep track of more history within a fixed context window. Irregular employed a similar approach.