Opening article
Hugging Face has launched AraGen, a benchmark and leaderboard for Arabic large language models that uses a new 3C3H evaluation measure. The measure scores responses on six dimensions—Correctness, Completeness, Conciseness, Helpfulness, Honesty, and Harmlessness—using an LLM-as-judge. AraGen’s leaderboard runs blind tests for three months, then releases the dataset and code before introducing a new private benchmark, aiming to reduce data contamination and keep evaluations current.
In the rapidly evolving landscape of large language models (LLMs), comprehensive and robust evaluation methodologies remain a critical challenge, particularly for low-resource languages. In this blog, we introduce AraGen, a generative tasks benchmark and leaderboard for Arabic LLMs, based on 3C3H, a new evaluation measure for NLG which we hope will inspire work for other languages as well.
3C3H Measure : The 3C3H measure scores a model's response and is central to this framework. It is a holistic approach assessing model responses across multiple dimensions - C orrectness, C ompleteness, C onciseness, H elpfulness, H onesty, and H armlessness- based on LLM-as-judge.
Dynamic Evaluations : AraGen Leaderboard implements a dynamic evaluation strategy, which includes three-month blind testing cycles, where the datasets and the evaluation code remain private before being publicly released at the end of the cycle, and replaced by a new private benchmark.
Arabic Evaluation Dataset : AraGen Benchmark offers a meticulously constructed evaluation dataset for Arabic LLM evaluation, combining multi-turn and single-turn scenarios, which tests the model capability across multiple domains and tasks.
We believe that AraGen addresses persistent issues of data contamination with its dynamic evaluation approach, preserving the benchmark's integrity. It also serves as the first application of a scalable, language-agnostic framework for a nuanced and fair model assessment, which represents an important effort in understanding LLM performance across diverse linguistic contexts and sets a new standard for comprehensive model benchmarking.
Evaluating large language models (LLMs) is a key challenge in AI research. While existing methodologies have improved our understanding of LLM capabilities, they often fail to comprehensively address both factuality —assessing a model's core knowledge—and usability —its alignment with human (end user) expectations. Current evaluation approaches can broadly be categorized into knowledge or factuality-based benchmarks and preference-based benchmarks.
By providing a language‑agnostic, dynamic benchmark that balances factual accuracy with usability, AraGen offers the AI industry a more comprehensive and fair way to assess LLM performance, especially for low‑resource languages.