Hugging Face released Transformers v5 with first-class support for Mixture-of-Experts models, introducing foundational elements like expert backends and dynamic weight loading. Building upon this release, NVIDIA introduced NeMo AutoModel, an open library designed to scale custom generative AI model training. According to the source, NeMo AutoModel incorporates Expert Parallelism, DeepEP fused all-to-all dispatch, and TransformerEngine kernels while maintaining API compatibility with Hugging Face Transformers through a subclassed AutoModelForCausalLM. Evaluations show that this combination achieves 3.4 to 3.7 times higher training throughput and 29 to 32 percent less GPU memory during the fine-tuning of MoE models compared to native Transformers v5, using the same from_pretrained API with no other code changes. The library enables full fine-tuning of frontier-scale models like the 550-billion parameter NVIDIA Nemotron across multiple nodes, while addressing single-node challenges such as memory pressure, communication overhead, and core operation efficiency for models like Qwen3 and Nemotron 3 Nano.
Accelerating Transformers Fine-Tuning with NVIDIA NeMo AutoModel
NVIDIA has introduced NeMo AutoModel, an open library built on top of Hugging Face Transformers v5 to accelerate the fine-tuning of Mixture-of-Experts models. By adding Expert Parallelism, DeepEP fused all-to-all dispatch, and TransformerEngine kernels while maintaining API compatibility, the library delivers significant speedups and memory reductions over native Transformers v5.
Why it matters
Efficiently fine-tuning frontier-scale Mixture-of-Experts models requires overcoming significant infrastructure bottlenecks like memory overhead and token routing communication. By integrating advanced parallelisms and kernels into standard Hugging Face workflows without breaking API compatibility, this development lowers the barrier to scaling large language models.