Transcript analysis for AI agent evaluations

At the AI Security Institute (AISI), we test AI agents using a broad suite of evaluations that assess their ability to autonomously complete complex tasks such as web browsing, data analysis, software engineering, and scientific research.

Most often, we report our results in the form of average pass rates; what fraction attempts across a set of tasks an agent was able to solve. This is a useful metric for understanding what AI agents are capable of – but can flatten important nuances if used in isolation. During our evaluations, we also generate thousands of transcripts, each containing the equivalents of dozens of pages of text. A transcript typically includes the initial task instruction, the agent's commentary messages, the tool calls the agent makes, and the outputs returned by those tools. Analysing transcripts can supplement average pass rates with detail on an agent's patterns of behaviour and failure modes.

In this post, we explain why transcript analysis is useful for AI evaluation, as well as providing some takeaways from our own analysis of almost 6,400 transcripts from recent testing exercises. You can find in-depth results from this transcript analysis in our case study .

Average pass rates are one of the key statistics that we report in our pre-deployment testing exercises . They help us measure the trajectory of capabilities in security-critical domains like chemistry, biology, and cybersecurity. However, they have several important limitations:

AISI is increasingly analysing transcripts from its evaluations in addition to reporting task outcome statistics. We’ve found profiling agent activity has helped us to guard against issues that would undermine our claims about agent capability and given us a richer understanding of agent task approaches and failure modes.

In an early transcript analysis, we examined the activity of AI agents based on 9 models, on a private suite of offensive cybersecurity tasks. In total, we evaluated agents on 71 capture-the-flag tasks (CTFs), ranging in difficulty from trivial to expert. The agent used approach called ReAct , which prompts a language model to generate both reasoning traces and action logs in an interleaved manner. This produced 6,390 samples, many with transcripts the length of an entire novel.