Opening article
Today, we’re thrilled to share that we’ve used Math-Verify to thoroughly re-evaluate all 3,751 models ever submitted to the Open LLM Leaderboard, for even fairer and more robust model comparisons! The Open LLM Leaderboard is the most used leaderboard on the Hugging Face Hub: it compares open Large Language Models (LLM) performance across various tasks. In the leaderboard, models would have to end their answers with a very specific string (following the Minerva-Math paper ): This therefore meant re-evaluating all submitted models since June… and it completely overhauled the top 20 models on the MATH subset of the leaderboard.
3 weeks ago, we showed how hard it is to correctly evaluate LLM performance on math problems, and introduced Math-Verify , a better solution to validate models on math (read more in the announcement )!
Today, we’re thrilled to share that we’ve used Math-Verify to thoroughly re-evaluate all 3,751 models ever submitted to the Open LLM Leaderboard, for even fairer and more robust model comparisons!
The Open LLM Leaderboard is the most used leaderboard on the Hugging Face Hub: it compares open Large Language Models (LLM) performance across various tasks. One of these tasks, called MATH-Hard, is specifically about math problems: it evaluates how well LLMs solve high-school and university-level math problems. It uses 1,324 highest difficulty problems (Level 5) from the Hendrycks MATH dataset spread across 7 topics (precalculus, prealgebra, algebra, intermediate algebra, counting/probability and number theory), using a 5-shot approach (the model is provided with 5 examples in the prompt to showcase how it should answer).
To start, a recurring issue was the inability of some models to follow the expected answer format from the examples: they outputted other sentences instead to introduce their answers. Since the format was not followed, answers were marked as wrong even if they were actually correct! (Which is an issue if what you’re interested in is “how good the model is at math” specifically).
In the leaderboard, models would have to end their answers with a very specific string (following the Minerva-Math paper ):
The leaderboard would then try to parse [ANSWER] with SymPy to convert it to a symbolic representation (and simplify the values if needed), before finally comparing it to the gold target.
The next step, converting [ANSWER] to the symbolic representation also presented some issues, this time linked to the SymPy parsing:
Fixing Open LLM Leaderboard with Math-Verify. Covered by Hugging Face on February 14, 2025.