Arabic Leaderboards: Introducing Arabic Instruction Following, Updating AraGen, and More

At Inception, we have been working to enhance AI model evaluations within the Arabic language context. Previously, we introduced AraGen , one of the first generative Arabic leaderboards, serving as a benchmark for evaluating Arabic LLMs on generative tasks.

As part of our ongoing efforts, we are excited to share the following updates:

The following sections provide details about each of these updates.

This allocation reflects the primary focus on question answering as the main use cases of any Language-Model/Chatbot/AI-Assistant, while still addressing other evaluation areas, particularly given the complexity of generating challenging queries in Arabic grammar and orthography.

One emergent trend is that the various dimensions are almost perfectly correlated. In most cases, correct answers are scored as both highly helpful and harmless, while most models fail to maintain this correlation for the conciseness dimension. This generally reflects the way we train these models today, where more verbose answers are often rewarded as more helpful. This trend has recently caught the attention of the research community, as exemplified by the release of OpenAI’s GPT-4.5 model. According to their use cases section , answers from GPT-4.5 are more concise than those from GPT-4, while still being equally helpful.

One of the core capabilities of large language models (LLMs) is their ability to understand and follow human instructions. This skill is crucial for building reliable chatbots, virtual assistants, and AI systems that do what users ask. Without strong instruction-following, a model might generate correct information but in the wrong format, ignore user-specified constraints, or produce unwanted content. Instruction-Following benchmark is standardized, objective way to measure a model's instruction adherence and compare models fairly to drive improvements.

Arabic-Leaderboards is a comprehensive and unified space for all Arabic evaluations and tasks. It is meant to serve as a central hub covering a broad spectrum of evaluations, for models across modalities. Currently, it has AraGen-03-25 and Arabic Instruction Following as live leaderboards. We plan to expand this space with more leaderboards and tasks for Arabic AI models across various modalities.

We invite interested contributors to reach out to us through the community tab or directly through email in order to discuss how to integrate their work/leaderboards as additional tabs into this space.

In December 2024, we introduced the AraGen Benchmark as the foundation for the AraGen Leaderboard. A key feature of this leaderboard is its dynamic nature, with evaluation datasets remaining private (blind testing) for three months to ensure fair and unbiased assessments. Adhering to the same philosophy, we are publicly releasing the AraGen-12-24 benchmark , along with all model responses evaluated by Claude-3.5-Sonnet following the 3C3H guidelines .