Opening article
Hugging Face has launched Judge Arena, a platform that lets users compare large language models (LLMs) as judges of natural language outputs. Participants run multiple models on test samples, vote for the judge that best matches their judgment, and the results populate a public leaderboard updated hourly with Elo scores. The arena currently includes 18 state‑of‑the‑art generative LLMs, both open‑source and proprietary, and will share anonymized voting data to aid the community.
LLM-as-a-Judge has emerged as a popular way to grade natural language outputs from LLM applications, but how do we know which models make the best judges ?
We’re excited to launch Judge Arena - a platform that lets anyone easily compare models as judges side-by-side. Just run the judges on a test sample and vote which judge you agree with most. The results will be organized into a leaderboard that displays the best judges.
Crowdsourced, randomized battles have proven effective at benchmarking LLMs. LMSys's Chatbot Arena has collected over 2M votes and is highly regarded as a field-test to identify the best language models. Since LLM evaluations aim to capture human preferences, direct human feedback is also key to determining which AI judges are most helpful.
We selected 18 state-of-the-art LLMs for our leaderboard. While many are open-source models with public weights, we also included proprietary API models to enable direct comparison between open and closed approaches.
Atla currently funds this out of our own pocket. We are looking for API credits (with no strings attached) to support this community effort - please get in touch at support@atla-ai.com if you are interested in collaborating 🤗
Review both judges’ evaluations and vote for the one that best aligns with your judgment
To avoid bias and potential abuse, the model names are only revealed after a vote is submitted.
Judge Arena focuses on the LLM-as-a-Judge approach, and therefore only includes generative models (excluding classifier models that solely output a score). We formalize our selection criteria for AI judges as the following:
By offering a transparent, data‑driven comparison of LLMs as judges, Judge Arena supports the development of more reliable and aligned evaluation tools, which is critical for advancing trustworthy AI systems.