Mixture of Experts (MoEs) in Transformers

Over the past few years, scaling dense language models has driven most progress in LLMs. From early models like the original ULMFiT (~30M parameters) or GPT-2 (1.5B parameters, which at the time was considered "too dangerous to release" 🧌), and eventually to today’s hundred-billion–parameter systems, the recipe was simple:

Scaling laws reinforced this trend, but dense scaling has practical limits:

If you're already familiar with MoEs and want to jump straight into the engineering work done in transformers, you can head directly to Transformers and MoEs .

This is what the Experts Backend system (introduced in PR #42697 ) addresses. The Experts Backend introduces a pluggable execution architecture that decouples expert computation from the model implementation. Instead of hardcoding one dispatch strategy inside each MoE model, the system allows expert layers to dynamically select a backend at runtime.

The decorator wraps expert classes and dispatches computation to the selected backend automatically.

A Mixture of Experts model keeps the Transformer backbone, but replaces certain dense feed-forward layers with a set of experts . An “expert” is not a topic-specialized module (e.g., "math expert", "code expert"). It is simply a learnable sub-network. For each token, a router selects a small subset of experts to process it.

Model capacity depends on total parameters, but inference speed depends on active parameters.

For example, take gpt-oss-20b . It has 21B total parameters, but uses 4 active experts per token, out of a total of 32 experts. Considering the shared components plus the active experts, this model uses ~3.6B active parameters per token. Running this model on an M3 Ultra Mac, which has a memory bandwidth of about 800 GB, we could estimate generation speed as ~ 800 / (3.6 * 2) in bfloat16 , where each parameter takes 2 bytes. This yields about 111 tokens per second . The actual performance number we get is ~115 tok/s, which is very close to the back-of-the-envelope calculation.

This super fast speed confirms the model works approximately as a 3.6B parameter one, but it has the same capacity (or quality) as a 21B parameter model.

Making MoEs first-class citizens in transformers means redesigning parts of the loading pipeline, execution model, and distributed abstractions, not just adding new model classes. We’ll focus on how the transformers library has evolved to support sparse architectures across:

Different tokens activate different experts, based on their hidden representations.

(Note: speed would be even faster if we used kernels for the native mxfp4 quantization the model uses).

Experts provide a structural boundary in the computation graph. Since different tokens engage different experts, we can parallelize across experts (we discuss this later in Expert Parallelism ).