A shared playbook for trustworthy third party evaluations

Independent, trusted third party evaluations play a critical role ⁠ in strengthening the safety ecosystem. These evaluations are conducted on frontier models to provide additional evidence for claims about critical capabilities and safety mitigations. In this post, we share lessons we’ve learned so far, and recommend approaches for designing evaluations that can validly assess frontier models that we hope help inform emerging standards in the space.

Earlier, many evaluations treated models like chatbots: the evaluation prompted a model as though it were a user asking a question, the model answered, and an evaluator judged the output. Today’s frontier models can do much more: they can use tools, keep track of information across many steps, and act within a larger workflow. This means that performance depends not only on the model, but also on the environment in which the task takes place, and on the setup that facilitates its actions. This surrounding setup, which we call the “harness,” can change key aspects of the system’s performance, including how it uses tools, keeps track of information, or recovers from mistakes.

This changes how evaluations need to be conducted, and what readers should look for in evaluation reports. In our view, the most useful reports explicitly describe two things beyond the result itself: First, they specify what claim the evaluation setup was designed to test, and second, they share the available evidence that the evaluation result is valid.

Safeguard testing can understate whether an attack can succeed, and how severe it could be, when not accounting for the resources available to attackers, including custom harnesses. In UK AISI's GPT‑5.5 cyber evaluation ⁠ (opens in a new window) , their expert red teaming found a universal jailbreak that elicited violative cyber content across the malicious queries OpenAI provided, including in multi-turn agentic settings. They used Codex to create a custom harness to strengthen the model’s attack performance: it embedded a reusable safeguard-bypass pattern into the interaction, preserved that pattern across turns and blocks, and applied it across the malicious cyber queries OpenAI provided. Safeguard testing should match the adversary. If the claim is about robustness to expert misuse, the test should evaluate the strongest credible end-to-end attack strategy under a defined budget, including any harness needed to preserve and reuse that strategy. Otherwise, the results risk miscalibration: they could support only a narrower claim about resistance to simpler prompting, could miss both how severe the attack becomes and its probability of success once the elicitation method is operationalized, and could also overstate how likely or severe a problem is if given too much budget.

There is a time and place for standardized harness comparisons, but evaluators should be explicit about why using a consistent set of harnesses is appropriate and what claim it can support. METR's time-horizon evaluation ⁠ (opens in a new window) is an example of a broader, appropriately fixed evaluation setup: it is designed to produce comparable results across the systems it evaluates. METR defines a common outcome, the typical duration for a human task at which an AI agent is predicted to succeed at a given reliability level. It applies a shared task suite, scoring method, fitting method, and a small set of reusable scaffolds such as Triframe and ReAct ⁠ (opens in a new window) within each batch of estimates reported together. When METR expanded the task suite and moved evaluation infrastructure from a framework called Vivaria to one called Inspect, it reported the change ( Time Horizon 1.1 update ⁠ (opens in a new window) ) and re-evaluated models under the new evaluation setup. That is the value of a standardized evaluation setup, including a consistent harness set: it can make readers confident that a difference in scores really reflects a difference between the systems being compared, rather than a change in the measurement setup.

We recommend that third party evaluation reports state what kind of claim their evaluation setup is meant to support; describe how closely what was tested reflects that broader claim; describe the harness choices that shaped the result; detail when those choices change between evaluations; and include supporting evidence to show how the result was produced and how well it generalizes to the claim.

Capability elicitation : Can a model plausibly produce the capability being evaluated?

Safeguard performance : How robust are the tested safeguards against the behavior or attack being evaluated?

Evaluation reports also need to explain how evaluators checked for effects that could impact the validity of a result. These include: