Hugging Face and the Allen Institute for AI have introduced BenchMIRT, a method for auditing large language model benchmarks at the level of individual prompts. Building on psychometric Item Response Theory, BenchMIRT uses multidimensional item response theory to estimate model strengths, question difficulty, and how well tasks distinguish performance. When trained on benchmarking results from 100 LLMs across 16 benchmarks containing over 34,000 questions, BenchMIRT independently recovered two dominant dimensions: safety and general reasoning. While it confirmed the intended focus for many evaluations, it revealed that certain safety benchmarks, such as BBQ and WMDP, align more strongly with general reasoning. Additionally, HarmBench demonstrated how a single benchmark can mix different signal types. The tool can also help optimize evaluations by identifying the most informative questions, showing that keeping only 50% or even 10% of benchmark questions can preserve capability measurements. Furthermore, BenchMIRT correctly predicted held-out question outcomes 79 percent of the time. The authors note limitations, as the training models were released by March 2025, meaning the analysis does not capture newer model generations.
BenchMIRT: What are LLM benchmarks actually measuring?
Researchers from Hugging Face and Allen Institute for AI have introduced BenchMIRT, a new method for auditing LLM benchmarks at the individual prompt level. Based on multidimensional item response theory, BenchMIRT helps separate overlapping signals like safety and general reasoning. The method was trained on benchmarking results from 100 LLMs across 16 benchmarks.
Why it matters
BenchMIRT helps the AI industry look beyond single benchmark scores to understand the specific underlying capabilities being measured by individual prompts, improving evaluation efficiency and interpretability.
Topic dossiers Hugging Face