📚 3LM: A Benchmark for Arabic LLMs in STEM and Code

Arabic Large Language Models (LLMs) have seen notable progress in recent years, yet existing benchmarks fall short when it comes to evaluating performance in high-value technical domains. Most evaluations to date have focused on general-purpose tasks like summarization, sentiment analysis, or generic question answering. However, scientific reasoning and programming are essential for a broad range of real-world applications, from education to technical problem-solving.

To address this gap, we introduce 3LM (علم) , a multi-component benchmark tailored to evaluate Arabic LLMs on STEM (Science, Technology, Engineering, and Mathematics) subjects and code generation. 3LM is the first benchmark of its kind, designed specifically to test Arabic models in structured reasoning and formal logic which are domains traditionally underrepresented in Arabic NLP.

3LM is made up of three datasets, each targeting a specific evaluation axis: real-world multiple-choice STEM questions (MCQs), synthetic high-difficulty STEM questions, and translated code generation tasks.

For Native STEM , we collected Arabic PDF sources and applied a dual OCR approach to recover both plain text and math formulas. Questions were extracted using LLM-based chunking and pattern recognition, followed by classification into MCQ format with randomized answer order. Final samples were reviewed by native Arabic speakers with STEM expertise to confirm answer validity and readability.

The Native STEM benchmark consists of 865 MCQs extracted from authentic Arabic educational content, including textbooks, worksheets, and exam banks for grades 8 through 12. Questions span five core subjects: Physics, Chemistry, Biology, Mathematics, and Geography.

Each question is annotated with metadata including domain and difficulty (on a 1–10 scale). The data was sourced using a pipeline that combined OCR (including LaTeX math parsing via Pix2Tex), LLM-assisted question-answer extraction, and manual review. This dataset provides a realistic testbed for evaluating factual and conceptual understanding in Arabic models using real educational materials.

To introduce greater challenge and diversity, we created a synthetic subset of 1,744 MCQs using the YourBench pipeline. This component draws from Arabic textbook text, which is chunked, summarized, and used as input to an LLM-driven question generation system. The result is a curated set of questions focused on mid-to-high difficulty reasoning, including conceptual, analytical, and application-based problems.