Opening article
SmolLM3 is a 3‑billion‑parameter transformer decoder model released by Hugging Face that claims to outperform Llama‑3.2‑3B and Qwen2.5‑3B while matching larger 4‑B models. The authors provide a full engineering blueprint, detailing architecture changes such as grouped‑query attention, NoPE rotary removal, intra‑document masking, and training recipe adjustments. The model was trained on 11.2 trillion tokens over 24 days on 384 H100 GPUs, with a three‑stage pretraining strategy and a mid‑training phase for long‑context and reasoning abilities, extending context to 64k tokens and enabling 128k inference via YARN.
Small language models are becoming increasingly important as users seek capable models that can be deployed efficiently. The community has produced a fascinating range of capable small models, each pushing the boundaries of what's possible at this scale. With SmolLM3, we're excited to contribute a new competitive fully open 3B model:
SmolLM3 sits in the efficiency sweet spot. Our 3B model outperforms Llama-3.2-3B and Qwen2.5-3B while staying competitive with larger 4B alternatives (Qwen3 & Gemma3). Beyond the performance numbers, we're sharing exactly how we built it using public datasets and training frameworks.
The complete recipe: We're releasing SmolLM3 with our engineering blueprint. It includes architecture details, exact data mixtures showing how we progressively boost performance across domains in a three-stage pretraining approach, and the methodology for building a hybrid reasoning model. Usually, achieving these results would require months of reverse engineering. Instead, we're providing the full methodology.
SmolLM3 both changed the architecture and data mixture over its predecessors. Let’s have a look at the architecture and training configurations first!
SmolLM3 follows a transformer decoder architecture with tied embedding similar to SmolLM2, building on Llama architecture with some key modifications optimized for efficiency and long context performance.
All these changes were validated through ablations using the same 3B architecture trained on 100B tokens from FineWeb-Edu, ensuring each modification either improved performance or maintained it while offering other benefits.
In addition to architecture changes we also ablated and improved the training recipe. Let’s have a closer look.
The chat template provides a default system message for both reasoning modes, along with a metadata section that includes the date, knowledge cut-off date, and current reasoning mode. Users can replace the default system message by providing one with the system role. The metadata section can be excluded by using the /system_override flag in the system prompt, offering flexibility for specific use cases.
Before diving into the training methodology, it's essential to establish how users interact with our dual-mode model. The chat template serves as the interface that enables seamless switching between reasoning and non-reasoning modes, and its design directly impacts both our training data format and model behavior. SmolLM3's chat template allows users to control the reasoning mode during a conversation. Users can activate reasoning or non-reasoning modes by including the /think and /no_think flags, respectively, in the system prompt. In non-reasoning mode, we pre-fill the model's response with empty think blocks, similar to Qwen3, to ensure direct answers without explicit reasoning.
SmolLM3 supports tool calling, and its chat template incorporates two distinct sections for tool descriptions: XML Tools and Python Tools. This specific categorization proved beneficial in our experiments for the model's accurate interpretation of tool definitions in each format.
Whether you're building your own models or want to understand what drives performance at this scale, this blueprint shows the engineering story behind competitive 3B performance.
Grouped Query Attention (GQA): We replaced multi-head attention with grouped-query attention using 4 groups. Our ablations on a 3B model trained with 100B tokens from FineWeb-Edu showed that GQA matches the performance of multi-head attention while significantly reducing the KV cache size during inference.
NoPE: We implemented NoPE from " RoPE to NoRoPE and Back Again: A New Hybrid Attention Strategy " (Yang et al., 2025), selectively removing rotary position embeddings from every 4th layer. This approach improves long context performance without affecting short context capabilities, as confirmed by our ablations.
Providing a fully open, reproducible recipe for a competitive 3‑B model lowers the barrier for researchers and developers to build efficient, multilingual, long‑context capable systems, accelerating innovation in the AI industry.