tokenizers v1: encode, decode and scaling, measured

Hugging Face has announced tokenizers v1, a major performance-focused refactor designed to address bottlenecks in machine learning pipelines. As models become faster and workloads scale through massive datasets, long inputs, and concurrent requests, tokenization can starve models of data if it fails to keep pace. According to Hugging Face, tokenizers v1 encodes text between 3 and 30 times faster than version 0.23 across ten model families using a single thread on an Apple M4 Max, scaling at 76 percent of linear across eight workers. The upgrade maintains exact output compatibility with v0.23, preserving token IDs, vocabularies, APIs, and merge ranks while supporting BPE, WordPiece, and Unigram models. Key performance improvements stem from several pipeline changes. For supported grammars, hand-written splitting functions replace general-purpose regex engines, utilizing CPU SIMD instructions and bitcannon bit streams to process bytes in parallel. A new thread-local cache maps pre-token bytes to token IDs to skip repeated merges for recurring words. Additionally, the BPE merge loop reuses caller-owned scratch buffers to eliminate repeated memory allocations, packs candidate pairs into 64-bit values for branchless comparisons, and processes batches of pre-tokens in a single model call. The release also incorporates ideas from open-source libraries and benefits from testing and patches contributed by IBM, NVIDIA, and the ExecuTorch team.