Opening article
mmBERT is a new massively multilingual encoder built on ModernBERT, trained on over 3 trillion tokens in 1,833 languages. It introduces a progressive language inclusion strategy, dynamic masking, and TIES merging, achieving strong performance on English GLUE, XTREME, and MTEB benchmarks while outperforming XLM‑R and matching English‑only models. The model also shows rapid learning of low‑resource languages during a final decay phase, surpassing larger LLMs on Faroese QA tasks.
This blog post introduces mmBERT , a state-of-the-art massively multilingual encoder model trained on 3T+ tokens of text in over 1800 languages. It shows significant performance and speed improvements over previous multilingual models, being the first to improve upon XLM-R, while also developing new strategies for effectively learning low-resource languages. mmBERT builds upon ModernBERT for a blazingly fast architecture, and adds novel components to enable efficient multilingual learning.
If you are interested in trying out the models yourself, some example boilerplate is available at the end of this blogpost!
mmBERT was trained on a carefully curated multilingual dataset totaling over 3T tokens across three distinct training phases. The foundation of our training data consists of three primary open-source and high-quality web crawls that enable both multilingual coverage and data quality:
mmBERT builds upon the ModernBERT architecture but introduces several key innovations for multilingual learning:
We use the same core architecture as ModernBERT-base with 22 layers and 1152 intermediate dimensions, but switch to the Gemma 2 tokenizer to better handle multilingual text. The base model has 110M non-embedding parameters (307M total due to the larger vocabulary), while the small variant has 42M non-embedding parameters (140M total).
Modern Architecture Benefits : The efficiency gains come from two main technical improvements:
Energy Efficiency : The combination of better throughput and modern architecture results in lower computational costs for inference, making mmBERT more practical for production deployments where multilingual support is needed at scale.
DCLM and Filtered DCLM provides the highest quality English content available, serving as the backbone for strong English performance (with the filtered data coming from Dolmino ). This dataset represents state-of-the-art web filtering techniques and forms a crucial component. Due to the high quality of this data, we use a signficantly higher proportion of English than previous generation multilingual encoder models (up to 18%).
FineWeb2 delivers broad multilingual web content covering over 1,800 languages. This dataset enables our extensive multilingual coverage while maintaining reasonable quality standards across diverse language families and scripts.
FineWeb2-HQ consists of a filtered subset of FineWeb2 focusing on 20 high-resource languages. This filtered version provides higher-quality multilingual content that bridges the gap between English-only filtered data and broad multilingual coverage.
The model demonstrates that a carefully staged multilingual training regime can deliver competitive or superior performance to larger LLMs while maintaining efficiency, informing future multilingual model design and deployment strategies.