Opening article
Transformers v5 introduces a major redesign of tokenizers, separating tokenizer architecture from trained vocabulary. The new design mirrors PyTorch’s separation of model architecture and weights, allowing tokenizers to be inspected, customized, and trained from scratch with less friction. The library now uses a clean class hierarchy with a Rust‑based backend for speed and a Python backend for legacy models. AutoTokenizer remains the recommended entry point for loading tokenizers. The change enables developers to create blank tokenizer architectures and train them on custom data, which was not possible in v4.
Transformers v5 redesigns how tokenizers work. The big tokenizers reformat separates tokenizer design from trained vocabulary (much like how PyTorch separates neural network architecture from learned weights). The result is tokenizers you can inspect , customize , and train from scratch with far less friction.
TL;DR: This blog explains how tokenization works in Transformers and why v5 is a major redesign, with clearer internals, a clean class hierarchy, and a single fast backend. It’s a practical guide for anyone who wants to understand, customize, or train model-specific tokenizers instead of treating them as black boxes.
For experts: If you're already familiar with the concepts and want to understand the changes in v5, go to v5 Separates Tokenizer Architecture from Trained Vocab
The tokenizer system in transformers forms a layered architecture:
The most significant change in Transformers v5 is a philosophical shift in how tokenizers are defined. Tokenizers now work like PyTorch's nn.Module : you define the architecture first, then fill it with learned parameters.
Worst of all, you couldn't create an empty tokenizer architecture. If you wanted to train a LLaMA-style tokenizer on your own data, there was no clean way to instantiate a "blank" LLaMA tokenizer and fill it with your vocabulary and merges. Tokenizers existed only as loaded checkpoints, not as configurable templates.
v5 treats tokenizer architecture (normalizer, pre-tokenizer, model type, post-processor, decoder) as distinct from trained parameters (vocabulary, merges). This mirrors how PyTorch separates model architecture from learned weights.
The Python backend is slower than the Rust backend. For most use cases, the Rust-backed TokenizersBackend is preferred.
Before diving into the changes, let's quickly cover what tokenization does and how the pieces fit together.
Language models don't read raw text. They consume sequences of integers usually called token IDs or input IDs . Tokenization is the process of converting raw text into these token IDs. (Try the tokenization playground here to visualize tokenization.)
Tokenization is a broad concept used across natural language processing and text processing generally. This post focuses specifically on tokenization for Large Language Models (LLMs) using the transformers and tokenizers libraries.
The redesign gives AI practitioners greater flexibility to customize tokenization pipelines, reducing reliance on black‑box pretrained tokenizers and enabling more efficient model training and deployment.