QIMMA, an Arabic LLM leaderboard, was created to address growing concerns about the reliability of existing Arabic benchmarks. The platform aggregates 109 subsets from 14 source benchmarks into a unified suite of over 52,000 samples spanning seven domains. Before any model is evaluated, QIMMA applies a rigorous multi‑stage validation pipeline. Two state‑of‑the‑art Arabic‑capable LLMs independently score each sample against a 10‑point rubric; samples scoring below 7/10 are eliminated or flagged for human review. Native Arabic speakers then make final judgments, especially for culturally sensitive content. The pipeline uncovered systematic quality issues such as mistranslated or mismatched gold answers, spelling and grammar errors, duplicate samples, and stereotype reinforcement. For code benchmarks, QIMMA refined Arabic problem statements while preserving task identifiers, reference solutions, and test suites. The evaluation framework relies on LightEval, EvalPlus, and FannOrFlop, and all prompts are in Arabic with six standardized template types. As of April 2026, QIMMA’s leaderboard lists 46 models, revealing a general size‑performance correlation with notable exceptions. The platform’s open‑source nature, native Arabic focus, systematic validation, code evaluation, and public per‑sample outputs distinguish it from other Arabic NLP evaluation resources. This effort ensures that reported scores truly reflect Arabic language capability, benefiting AI developers, researchers, and stakeholders working with Arabic‑speaking populations.
QIMMA قِمّة ⛰: A Quality-First Arabic LLM Leaderboard
QIMMA (Arabic for "summit") is an Arabic LLM leaderboard that validates benchmark quality before evaluation. It consolidates 109 subsets from 14 source benchmarks into over 52,000 samples across seven domains, and applies a multi‑stage validation pipeline involving two Arabic‑capable LLMs and native human reviewers. The platform is open source, uses predominantly native Arabic content, and publishes code evaluation and per‑sample inference outputs.
Why it matters
By ensuring benchmark quality, QIMMA provides more reliable assessments of Arabic LLMs, which is critical for developers and researchers targeting Arabic‑speaking markets.