Opening article
Hugging Face released Ettin, a suite of paired encoder‑only and decoder‑only language models trained with identical data, architecture, and recipes. The models, ranging from 17 M to 1 B parameters, use a three‑phase training approach on 2 T tokens and outperform ModernBERT encoders and baseline decoders such as Llama 3.2 1B and SmolLM2. Ettin enables direct comparison of the two architectures, revealing that encoders excel at classification and retrieval while decoders lead on generation tasks.
What would happen if you took the ModernBERT recipe and applied it to a decoder-only model? Turns out, a state-of-the-art decoder language model that beats Llama 3.2 1B and SmolLM2!
We introduce a new open-data training recipe to reproduce the encoder-only ModernBERT model (and actually beat it!). We then apply the exact same recipe to decoder-only models. For the first time, we have two state-of-the-art models trained in the same setup but with two different training objectives: masked language modeling (MLM), and causal language modeling (CLM).
This blog post introduces Ettin , the first suite of SoTA paired encoder-only and decoder-only models (17M-1B params) trained with identical data (2T tokens), architecture, and training recipes. Ettin enables true apples-to-apples comparisons between architectures and delivers state-of-the-art performance for open-data models in both categories. We then further explore whether it is possible to get a competitive encoder starting from the decoder and vice-versa.
While decoder models have seen rapid innovation, encoder model development had stagnated – until recently, with efforts like ModernBERT modernizing them. But which architecture is better? Previous comparisons between encoders and decoders used different datasets, architectures, and training recipes, so it was hard to tell.
We train six different sizes, ranging from 17M to 1B parameters. This allows us to test the effects of scale, and provides a wide variety of models for you to use! No matter if you need a blazing fast on-device model or a powerful but slower model, we got you covered!
This suggests the architecture choice matters fundamentally, not just the training objective.
If you are interested in trying out the models, some boilerplates are available at the end of this blogpost!
The LLM community has largely converged on decoder-only models like GPT, Llama, and Qwen. Their generative capabilities are impressive, but this focus is detracting attention from other categories, such as encoder-only models like BERT.
However, encoder BERT-like models remain the workhorses of production systems for classification, retrieval, and embedding tasks. They're faster, more memory-efficient, and often more accurate for discriminative tasks. The key difference lies in their attention patterns:
By providing a controlled, apples‑to‑apples comparison, Ettin clarifies the inherent strengths of encoder versus decoder architectures, guiding model selection for specific AI workloads.