Mixture of Experts (MoEs) in Transformers

Mixture of Experts (MoE) models keep the Transformer backbone but replace certain dense feed‑forward layers with a set of learnable sub‑networks called experts. For each token a router selects a small subset of experts to process it, so different tokens activate different experts based on their hidden representations. The overall model capacity depends on the total number of parameters, but inference speed depends on the active parameters. For example, the gpt‑oss‑20b model has 21 B total parameters but uses only 4 active experts per token out of 32, resulting in about 3.6 B active parameters per token and a measured generation speed of roughly 115 tokens per second on an M3 Ultra Mac. MoEs can outperform dense counterparts given a fixed training FLOP budget because they provide a structural boundary that allows parallelization across experts. Recent open MoE releases include Qwen 3.5, MiniMax M2, GLM‑5, Kimi K2.5, and Mixtral‑8x7B, with the trend accelerating after DeepSeek R1 in January 2025. Closed labs also use MoEs; ChatGPT is rumored to employ a sparse architecture. The Hugging Face transformers library has evolved to support MoEs by redesigning the weight‑loading pipeline. A new generic WeightConverter allows checkpoints, where each expert is serialized independently, to be converted into a contiguous tensor layout suitable for modern MoE kernels such as grouped GEMMs and fused implementations. The loader scans checkpoint keys once, matches them against converter patterns, and schedules conversions, enabling parallel shard loading and async materialization. The Experts Backend introduces a pluggable execution architecture that decouples expert computation from the model implementation, offering eager, batched_mm, and grouped_mm dispatch strategies. These developments make MoEs first‑class citizens in transformers, improving loading speed, enabling per‑expert quantization, and providing efficient runtime dispatch for large‑scale language models.