Opening article
Since June 2024, Hugging Face has evaluated over 3,000 open language models on the Open LLM Leaderboard, measuring both performance and CO₂ emissions during inference. The analysis shows that community‑fine‑tuned models often consume less carbon than their official counterparts, especially for large‑parameter models. The study highlights differences in verbosity and output length between base, official, and community models, which affect energy use.
Since June 2024, we have evaluated more than 3,000 models on the Open LLM Leaderboard, a worldwide ranking of open language models performance. Even though we’re trying to run evaluations without wasting resources (we use the spare cycles of our cluster, in other words the GPUs which are active but waiting between jobs), this still represents quite a big amount of energy spent for model inference!
In the last year, people have become more and more aware that using large language models (LLMs) to generate text has a significant environmental impact, beyond the already important impact of training. Recent research (see the Towards Greener LLMs article) highlights the challenges of managing resources efficiently at inference due to dynamic and diverse workloads.
By integrating carbon emission estimates into the Open LLM Leaderboard, we aim to provide transparency to users about the carbon impact of various model evaluations and hopefully encourage model creators to balance performance with environmental responsibility.
Since we wanted to look at general trends, we only considered the most frequent model architectures, and models for which we had the parameter count.
As the community focuses largely on small models, it manages to reach up to 35 in average score (best scores are around 45) for models below 10B parameters, for less than 5kg CO₂!
Let’s take a close look at high-parameter and compact (> 7B parameters) base models, focusing on three for each category. We will investigate the emissions for each base model itself, for other official fine-tunes, including the official instruct versions, and community fine-tunes.
We were curious to explore the CO₂ emissions associated with model inference and to identify any emerging trends in the data. Along the way, we saw a few predictable patterns but also discovered some surprising insights, like community fine-tunes being more carbon-efficient in general!
💡 Note: We’ve created a Colab notebook containing all the data and findings discussed here. This notebook allows you to explore the results, run the analyses yourself, and even adapt the code to investigate further questions
Let's take a closer look at how we calculate the CO₂ emissions generated during model inference.
CO₂ Emissions and Models Performance: Insights from the Open LLM Leaderboard. Covered by Hugging Face on January 9, 2025.