A structured protocol for elicitation experiments

Identifying dangerous AI capabilities requires rigorous evaluation of models at the upper limit of their abilities—a process that is far from straightforward. There are many ways to enhance the performance of a model after it has been trained, from carefully crafted prompts to external tool access. We need evaluation practices that uncover the full range of what models can be used to achieve.

These practices fall under the umbrella of elicitation , which plays a critical role in safety and security work. Done well, it helps us avoid underestimating what models can do, thereby improving the accuracy of and confidence in claims related to risk thresholds. This refined assessment directly informs how we test models at AISI and supports the UK government’s understanding of AI-related risks.

But elicitation is hard to get right. It’s part science, part craft—and current methods are often inconsistent and poorly transferable across models and tasks. As a result, it is difficult to accumulate insights or apply lessons across different contexts.

To support this work, we have developed a structured elicitation protocol —a checklist of best practices intended to guide rigorous and informative experimentation. The protocol is designed to promote:

Over the past two months the AISI Science of Evaluations Team has standardised elicitation experiment practices across our workstreams to ensure experiments are more reproducible, comparable, and analysable. This blog explains why capability elicitation is so important and summarises our approach.

Capability elicitation experiments are designed to unlock or enhance a model’s latent ablities after it has been trained, in order to best understand its capability profile. Techniques include:

All of these methods encourage the model to generate more accurate responses, as well as follow more effective and efficient sequences of actions, compared to a simple query.