Optimal stopping: spending evaluation compute where it counts
We introduce optstop, an open-source tool for LLM evaluations that keeps running where uncertainty is high, and stops where estimates are precise or stable enough. Broadly, optstop aims to reduce unnecessary compute use during evaluations by stopping runs when estimates of a model’s performance are precise enough. During an evaluation run, a model is paired with one or more benchmarks - a structured set of tasks, used to estimate a model’s underlying capability. The number of tasks, multiplied by the number of repetitions, gives you the total number of trials (single runs of the model), or the sample size of the evaluation.
The Core Development
As the capabilities of frontier AI models continue to advance, the evaluations we use to test and measure these capabilities must keep pace. Evaluations that elicit a model’s true capabilities are increasingly becoming longer and more expensive to run, while rapid model development and release cycles leave less time to run them.
AISI has previously found that fixed-budget evaluations can systematically underestimate frontier agentic capability , especially for newer models. Restricted-budget benchmark scores can make model comparisons unequal, lead decision-makers to underestimate agent capabilities, and obscure the true scale of risks. However, the scale of the compute budget needed to properly elicit capabilities in evaluations increasingly poses a practical issue for many evaluators - some frontier evaluations now require hundreds of millions of tokens.
At this scale, every unnecessary or wasted step in an evaluation has a meaningful opportunity cost. Evaluators therefore need efficient methods that use resources selectively, without weakening the statistical basis of an evaluation’s result, and without compromising the ability of an evaluation to elicit models’ actual capabilities.
Real-World Use Cases & Applications
We have been exploring optstop’s use cases at AISI as one potential method to help tackle evaluation bottlenecks, and plan to develop and release more tools like it in future.
Strategic Perspective & Ecosystem Impact
AISI’s mission is not only to evaluate the capabilities of frontier models, but also to innovate new and better tooling, methods, and frameworks to ensure evaluations remain fit for purpose – sharing solutions where we can, to benefit the wider community of evaluators.
One such solution we have developed to help tackle the challenge of evaluation efficiency is optstop, an open source Python package that integrates with our Inspect evaluation framework. Broadly, optstop aims to reduce unnecessary compute use during evaluations by stopping runs when estimates of a model’s performance are precise enough. During testing, we found that under every condition we set, optstop saved between 57% and 97% of planned runs in evaluations.
This blog will discuss the specific measurement problem that optstop addresses – and we explain how it works, how evaluators can use it, and what we found when running it in initial tests.
Why it matters
We introduce optstop, an open-source tool for LLM evaluations that keeps running where uncertainty is high, and stops where estimates are precise or stable enough Reported by AISI Blog on August 27, 2026.