Community Evals: Because we're done trusting black-box leaderboards over the community

Hugging Face is introducing a new evaluation framework that decentralizes the reporting of benchmark scores across its Hub. Benchmark datasets can register as leaderboards, automatically aggregating results from model repositories and displaying them on dataset cards. Models will store their evaluation scores in .eval_results/*.yaml files, which appear on the model card and feed into the benchmark leaderboards. Community members can submit evaluation results for any model via pull requests; these community scores are shown without waiting for model authors to merge or close them. The platform uses Inspect AI‑format eval.yaml files to define evaluation specifications, ensuring reproducibility. Verified badges will indicate that results can be reproduced. All scores will be exposed through Hub APIs, enabling the creation of curated leaderboards and dashboards. The initiative does not replace traditional benchmarks but complements them, aiming to make evaluation data more transparent and accessible. The feature is currently in beta and open for community feedback.