International joint testing exercise: Agentic testing

Jointly conducted by participants across the International Network, including representatives from Singapore, Japan, Australia, Canada, European Commission, France, Kenya, South Korea and the United Kingdom.

Participants from across the International Network, including representatives from Singapore, Japan, Australia, Canada, the European Commission, France, Kenya, South Korea and the United Kingdom have come together to align their approaches to agentic evaluations.

This is the third exercise, building on insights from two earlier joint testing exercises conducted by the Network in November 2024 and February 2025. The objective of these exercises is to allow the Network to further refine best practices for testing advanced AI systems.

The exercise was split into two strands— common risks: leakage of sensitive information and fraud (led by Singapore AISI) and cybersecurity (led by UK AISI). A mix of open and closed weights models were evaluated against tasks from various public agentic benchmarks.

To do this, Network participants from the UK, EU AI Office and Australia ran evaluations on two open-source models, anonymised in this document as Model E and Model F. We used two agentic cybersecurity capability benchmarks: Cybench and Intercode . We ran baseline evaluations on the two benchmarks to ensure consistent set ups across the AISIs, using a default temperature of 0.7, 10 samples per task, and 2.5 million token limits per task attempt. Each AISI then varied different parameters to assess the impact on agent capabilities and behaviours. We ran multiple variations of the parameters listed below, and then assessed their impact on the model’s ability to complete the task, as well as any changes in agent behaviour:

1. Run quick sweeps on a handful of representative tasks to identify optimal parameter settings : While the performance of the models tested in this exercise did not change significantly as a result of some variables being altered, they responded differently to temperature and maximum number of attempts. Model F’s accuracy declined as temperature rose, whereas Model E was largely unaffected. Parameters like this should ideally be optimised for each model being tested ahead of full evaluation.

The goal for this third exercise was to advance the science of AI agent evaluations and work towards building common best practices for testing AI agents . “LLM agents”, the focus of this exercise, are a type of agentic system that utilises a base LLM model to automatically reason through problems, plan a course of action, use tools and execute tasks. The rapid rise of autonomous AI systems and advancements in agent capabilities are introducing new risks due to reduced oversight over their real world interactions. Yet, agent testing remains nascent and is still a developing science. As AI agents are starting to be deployed globally, it is also important that these agents handle different languages and cultures accurately and securely.

Given the nascency of agentic testing, our primary focus was understanding the methodological issues in conducting agentic testing, rather than examining test results and model capabilities. Previous testing exercises have demonstrated that “small methodological changes can have a large impact on testing outcomes”. Our goal, therefore, was to inform and refine future evaluation efforts.

Traditional evaluation methods have proven insufficient for providing reliable insights into the complexities of autonomous agent behaviour, highlighting the need for novel approaches. To address this gap, participating countries contributed their collective technical and linguistic expertise. This collaboration marks an important step forward, as participants work together to advance the science of agentic evaluations.