Opening article
Inside VAKRA: Reasoning, Tool Use, and Failure Modes of Agents
Inside VAKRA: Reasoning, Tool Use, and Failure Modes of Agents — announced via Hugging Face on April 15, 2026. This capability includes 2,077 test instances across 54 domains, requiring the use of tools from the SLOT-BIRD and SEL-BIRD collections ( Elder et al., 2026 ). Each domain is restricted to one tool collection, and tasks involve chaining 1–12 tool calls to arrive at the final answer. This task requires selecting the correct APIs from the domain-specific tool set (as shown in the example in Figure 1 ). Capability 4 includes 644 instances across 41 domains and is also built on the REST-BIRD API collection.
Model Size
120B
Dense ParametersDeployment Platform
Hugging Face
Cloud InfrastructureCategory
Products
Industry DomainThe Core Development
VAKRA Dataset | LeaderBoard | Release Blog | GitHub | Submit to Leaderboard
We recently introduced VAKRA , a tool-grounded, executable benchmark for evaluating how well AI agents reason and act in enterprise-like environments.
Unlike traditional benchmarks that test isolated skills, VAKRA measures compositional reasoning across APIs and documents , using full execution traces to assess whether agents can reliably complete multi-step workflows.
How It Works & Technical Architecture
The Business Intelligence (BI) API capability contains two sets of APIs, from the SLOT-BIRD and SEL-BIRD tool collections. The SEL part of this benchmark had 600 samples, while the SLOT part of the benchmark had 1477 samples. These two collections are grouped under the BI API capability, but have slightly different characteristics. The SLOT-BIRD collection has a smaller number of generic tools with a large number of parameter values to fill, while the SEL-BIRD collection has a larger set of tools and fewer parameters per tool. This focus is reflected in the relative errors made by models using these two tool collections.
Deployment & Availability
Tool-Sequence Comparison Due to the presence of an executable environment, agents can explore the environment and sometimes return the answer by invoking a different set of APIs than the ones identified by us. In order to support alternative but valid tool invocations and reasoning paths, correctness is assessed by executing each predicted tool and comparing the set of tool responses against those from the ground truth (rather than enforcing strict step-level matching).
Strategic Perspective & Ecosystem Impact
VAKRA provides an executable environment where agents interact with over 8,000+ locally hosted APIs backed by real databases spanning 62 domains , along with domain-aligned document collections. Tasks can require 3-7 step reasoning chains that combine structured API interaction with unstructured retrieval under natural-language tool-use constraints.
As can be seen below, models perform poorly on VAKRA - in this blog, we include additional dataset details about the tasks in VAKRA and present an analysis of failure modes we observed on different tasks.
As shown below, the VAKRA benchmark comprises of four tasks, each testing a different set of capabilities .
Why it matters
Inside VAKRA: Reasoning, Tool Use, and Failure Modes of Agents — announced via Hugging Face on April 15, 2026 Reported by Hugging Face on April 15, 2026.