Opening article
Hugging Face has released the Falcon‑H1 series, a set of six open‑source language models ranging from 0.5B to 34B parameters. The models use a hybrid architecture that combines Transformer attention with a State Space Model (SSM) to achieve long‑context memory and efficient inference. Each size is available in base and instruction‑tuned variants, and all models are licensed under Apache 2.0.
Today, we are proud to introduce the Falcon-H1 series, a collection of six open-source models ranging from 0.5B to 34B parameters, each available in both base and instruction-tuned variants. At the core of these models lies a hybrid architecture that combines the strengths of the classical Transformer-based attention mechanism with the State Space Model (SSM), known for its superior long-context memory and computational efficiency. This architectural innovation is further enhanced by fundamental advancements in training dynamics and data utilization, enabling Falcon-H1 models to deliver uncompromised performance that rivals the top Transformer-based models across all covered size tiers.
In this release, we feature six open-weight models: 0.5B, 1.5B, 1.5B-Deep, 3B, 7B, and 34B, along with their instruct versions. All our open-source models are with a permissive license based on Apache 2.0.
Hybrid Architecture (Attention + SSM): We combine attention and Mamba-2 heads in parallel within our hybrid mixer block. Importantly, the amount of attention and mamba heads can be adjusted independently, allowing for an optimal attention/SSM ratio. This hybrid design enables faster inference , lower memory usage , and strong generalization across tasks.
When embarking on the Falcon-H1 series development, we chose to fundamentally rethink the training approach. While the field of LLM development has converged on many established practices that reliably produce strong models, these conventions were primarily validated on classical transformer architectures. The shift from pure attention mechanisms to a hybrid attention-SSM design represents a significant architectural change, making it uncertain whether these standard practices would remain optimal.
The hybrid attention-SSM models have a larger configuration space of all the parameters that define the model architecture. Our goal was to probe each of these configuration parameters to check its impact on model performance and efficiency. As a result, we reveal regions of model configuration space with an increased performance at a mild efficiency cost. We can roughly divide the hybrid model configuration space in the following 4 blocks:
On top of that, we made the next step into inner workings behind μP to further boost the model performance. In a nutshell, each component of the model “wants” to train at its own intensity, and that intensity depends on the size of the component. μP scaling rules take into account this dependence through so-called ``μP multipliers'' to enable optimal hyperparameter transfer. However, classical μP uses trivial multipliers of 1 at the base model size, which corresponds to a nasusmption that intensity of all components are already optimal at the base size. We discard this assumption and tune the multipliers at the base model size. Specifically, we have divided model parameters into 35 fine-grained groups and performed a joint optimization of the respective 35 multipliers.
We have observed that many aspects of the training dynamics are linked together under a common theme of noise interpretation and control. This includes learning rate and batch size schedules, scaling of the learning rate with batch size, and the behavior of parameter norms. In particular, we have found the parameter norms to be mostly determined by the training hyperparameters rather than the model fitting the data. To take this into account, we have included weight decay, a hyperparameter that primarily controls parameter norms, into both the training schedule and μP multipliers.
256K Context Support: Falcon-H1 models support up to 256K context length , enabling applications in long-document processing, multi-turn dialogue, and long-range reasoning.
Wide Range of Model Sizes: Available in six scales —0.5B, 1.5B, 1.5B-deep, 3B, 7B, and 34B—with both base and instruction-tuned variants, suitable for everything from edge devices to large-scale deployments.
Multilingual by Design: Supports 18 languages natively, including Arabic (ar), Czech (cs), German (de), English (en), Spanish (es), French (fr), Hindi (hi), Italian (it), Japanese (ja), Korean (ko), Dutch (nl), Polish (pl), Portuguese (pt), Romanian (ro), Russian (ru), Swedish (sv), Urdu (ur), and Chinese (zh) — with scalability to 100+ languages , thanks to our multilingual tokenizer trained on diverse language datasets.
Compact Models, Big Performance: Falcon-H1-0.5B delivers performance on par with typical 7B models from 2024 , while Falcon-H1-1.5B-Deep rivals many of the current leading 7B–10B models . Each Falcon-H1 model is designed to match or exceed the performance of models at least twice its size , making them ideal for low-resource and edge deployments without compromising on capability.
The Falcon‑H1 series demonstrates that hybrid attention‑SSM designs can match top Transformer models while offering longer context and lower resource usage, which is critical for scalable AI deployments.