Gaia2 and ARE: Empowering the community to study agents

In an ideal world, AI agents would be reliable assistants. When given a query, they would easily manage ambiguity in instructions, construct step-by-step plans, correctly identify necessary resources, execute those plans without getting sidetracked, and adapt to unexpected events, all while maintaining accuracy and avoiding hallucinations. However, developing agents and testing these behaviors is no small feat: if you have ever tried to debug your own agent, you’ve probably observed how tedious and frustrating this can be. Existing evaluation environments are tightly coupled with the tasks they evaluate, lack real-world flexibility, and do not reflect the messy reality of open-world agents: simulated pages never fail to load, events don’t spontaneously emerge, and asynchronous chaos is absent.

That’s why we’re very happy to introduce Gaia2, the follow-up to the agentic benchmark GAIA, allowing analysis of considerably more complex behaviors. Gaia2 is released with the open Meta Agents Research Environments (ARE) framework to run, debug and evaluate agents. ARE simulates complex real world-like conditions and can be customized to further study agents behaviors. Gaia2 dataset is released under CC by 4.0 license, and ARE under MIT license.

GAIA is an agentic benchmark published in 2023, with 3 levels of information retrieval questions requiring tools, web browsing, and reasoning to solve. In 2 years, the easiest levels have become too easy for models, and the community is coming close to solving the hardest questions, so it was time for an entirely new and harder agent benchmark!

Where GAIA was read-only, Gaia2 is now a read-and-write benchmark, focusing on interactive behavior and complexity management. Agents are now evaluated not only on search and retrieval, but also on instruction following over ambiguous or time-sensitive queries, in a noisy environment with controlled failures - reflecting real-world conditions more than any other simulated environment. We want to test how agents manage tools or APIs that sometimes do not work, plan successions of actions with very specific time frames, and adapt to new events - a whole new range of complexity!

For Gaia2, we created a smartphone mock-up environment, simulating what a human would use in their daily life. It contains real-world applications such as messaging (Email), utilities (Calendar, Contacts, Shopping, a FileSystem, …), and a chat interface to talk to the agent. All applications are also accessible to the agents through tool calling. Last but not least, the demo also contains a simulated persona’s history of conversations and app interactions.

Some capabilities appear to be already close to solved by the best models: execution of simple tool calls and instruction following ( execution ), and overall search (as we could have guessed from current results on GAIA). The ambiguity, adaptability, and noise splits remain challenging for now for all models, and it’s interesting to see that performance on what were considered complex agentic tasks (instruction following and search) is not a good proxy for performance on closer-to-real-world tasks. Last but not least, the hardest split for all models at the moment is the time one: it’s very hard at this moment for models to correctly handle time-sensitive actions (though this could likely be mitigated by the use of specialised tools and better temporal reasoning). Detailed analysis of these results can be found in the paper.

Here comes Gaia2, a follow up to GAIA, going way beyond it in terms of capabilities studied!

To do this, we use the following task groups (thanks to 1000 brand new human-created scenarios):

In the spirit of GAIA, scenarios do not require specialized knowledge: humans should in principle be able to get 100%, which allows easy debugging for model developers.