VAKRA, introduced by Hugging Face, is a tool‑grounded, executable benchmark designed to assess how well AI agents reason and act in enterprise‑like environments. Unlike traditional benchmarks that test isolated skills, VAKRA measures compositional reasoning across APIs and documents, using full execution traces to determine whether agents can reliably complete multi‑step workflows. The benchmark provides an executable environment where agents interact with over 8,000 locally hosted APIs backed by real databases spanning 62 domains, along with domain‑aligned document collections. Tasks require 3–7 step reasoning chains that combine structured API interaction with unstructured retrieval under natural‑language tool‑use constraints. VAKRA comprises four tasks, each testing a different set of capabilities: tool chaining with the SLOT‑BIRD and SEL‑BIRD collections, REST‑API selection with the REST‑BIRD collection, multi‑hop reasoning, and multi‑source, multi‑turn interactions that include document indices and tool‑usage policies. Models perform poorly on VAKRA, and the authors provide additional dataset details and an analysis of failure modes observed across tasks. The evaluation framework is execution‑centric, assessing not only final outputs but also the full tool‑execution trajectory, including tool calls, inputs, and intermediate results. The VAKRA Evaluator operates over two key inputs for each sample: a predicted final response and the corresponding tool‑call trajectory. The evaluation follows a waterfall‑style pipeline, where later stages are conditioned on earlier success, and correctness is assessed by executing each predicted tool and comparing the set of tool responses against those from the ground truth. This benchmark highlights the challenges of building agents that can navigate complex tool ecosystems and reason across multiple sources, providing a realistic testbed for advancing AI tool integration capabilities.
Inside VAKRA: Reasoning, Tool Use, and Failure Modes of Agents
VAKRA is a tool‑grounded, executable benchmark released by Hugging Face to evaluate AI agents’ reasoning and tool use in enterprise‑like settings. It features over 8,000 locally hosted APIs across 62 domains and includes document collections, requiring agents to perform 3–7 step reasoning chains that combine structured API calls with unstructured retrieval. The benchmark contains four tasks, each testing different capabilities such as tool chaining, REST‑API selection, multi‑hop reasoning, and multi‑source, multi‑turn interactions. Models perform poorly on VAKRA, and the authors analyze failure modes across tasks.
Why it matters
The benchmark forces agents to demonstrate coherent, multi‑step workflows and correct tool usage, highlighting gaps in current models and guiding future research in practical AI tool integration.