Modular LLMs at scale: how FlexOlmo is helping to pool national expertise without pooling sensitive data

When we released FlexOlmo last year, we wanted to show that a language model doesn't have to be a monolith. Different teams could train their own pieces – specialized modules called experts – in isolation and merge them into a shared model, without ever pooling the data underneath.

A project out of Denmark, Danish Foundation Models (DFM) , used FlexOlmo as the cornerstone for an architecture of their own. DFM develops open language models for the Danish language on the premise that models for lower-resource languages will fall behind unless independent efforts step in from outside well-resourced commercial labs. The institutions that would benefit most from Danish-language models – hospitals, universities, public-sector organizations, and smaller companies – often hold data that they can't share, whether for regulatory or proprietary reasons. Yet that data is exactly what's needed to train the models that would serve them.

"We envisioned a modular system whereby national initiatives like ours can independently train on their respective corpora, and then bring those independently trained models together," says Jacob Nielsen, an Industrial PhD Fellow at Ordbogen A/S and a researcher at the University of Southern Denmark's (SDU) OdenseNLP lab, a research group at SDU led by Peter Schneider-Kamp and Lukas Galke Poech and part of DFM. “More broadly, we aim to contribute to modular multilingual models that are beneficial on an international scale.”

In FlexMoRE, some experts are full-size, but most are replaced with much smaller versions called low-rank adapters—compact approximations of what a full-size expert learned using far fewer parameters. The size of each adapter is set by a value called its rank, and the team found that the best rank depends on what the expert is being asked to do: reasoning-heavy tasks, like working through a multi-step math problem, need higher ranks to preserve performance, while knowledge work, which draws on facts the model has learned, can use lower ones.

Because these two dominant task types – reasoning and knowledge – have different needs, FlexMoRE can shrink the overall model without losing capability. In its best configuration, FlexMoRE outperforms a FlexOlmo-style baseline of full-size experts while using less than one-third the parameters.

“It’s very exciting to see this Danish national project picking up our FlexOlmo architecture and adapting it for their important project,” says Sewon Min, an Assistant Professor in EECS at UC Berkeley, a Research Scientist at Ai2, and a co-author of the FlexOlmo paper. “We’re seeing growing momentum around modular training, both internally and across the broader research community, suggesting that decentralized and distributed training of foundation models is not merely a conceptually elegant idea, but a practical necessity. As frontier models become more and more costly to train and deploy, solutions like this become even more critical to make sure the immense benefits that come from AI systems are not consolidated in just a few hands.”

"The line of work around FlexOlmo and FlexMoRE has established that a separate-training joint-inference paradigm can be successful, and that it can be efficient," Nielsen says. "We believe that this is the core advantage of the modular architecture of independently trained experts."

Nielsen and a team of collaborators at OdenseNLP built FlexMoRE, which preserves FlexOlmo's modular structure but shrinks the model enough to run on consumer hardware. It addresses a limit DFM ran into with FlexOlmo: in the original framework, each expert is the size of a full standalone model, which works when there are only a handful of experts but doesn't scale easily. As more groups contribute their own experts, the combined system grows too large to run on the kinds of machines DFM's partners often have available.

"FlexMoRE significantly reduces FlexOlmo's memory demands while preserving performance across almost all categories, allowing a broader audience to benefit from modular models,” says Nielsen.

DFM covers the full stack of Danish-language AI: open training corpora, evaluation infrastructure, and a series of openly licensed Danish language models, all built to comply with the EU AI Act and GDPR. Its end goal is a system where a user can download just the experts they need – a few languages, plus a few domains – and run the combined model on their own hardware.