Opening article
Hugging Face’s Xet team has introduced a chunk‑based upload and download system that reduces data transfer times by up to 2‑3× for AI builders. By combining content‑defined chunking (CDC) with block‑level aggregation and a key‑chunk index, the system limits the number of network requests and database operations needed to deduplicate large repositories. The approach has already cut the storage footprint of a 191 GB quantized model set from 191 GB to about 97 GB, saving roughly 94 GB and cutting upload time by about four hours at 50 MB/s. The team plans to roll out Xet‑backed repositories in the coming weeks and months, aiming to make file transfers “invisible” for contributors.
Content-defined chunking (CDC) plays a central role in enabling deduplication within a Xet-backed repository . The idea is straightforward: break each file’s data into chunks, store only unique ones, reap the benefits.
In practice, it's more complex. If we focused solely on maximizing deduplication, the design would call for the smallest possible chunk size. By doing that, we’d create significant overheads for the infrastructure and the builders on the Hub.
On Hugging Face's Xet team , we're bringing CDC from theory to production to deliver faster uploads and downloads to AI builders (by a factor of 2-3x in some cases). Our guiding principle is simple: enable rapid experimentation and collaboration for teams building and iterating on models and datasets. This means focusing on more than just deduplication; we’re optimizing how data moves across the network, how it’s stored, and the entire development experience.
Imagine uploading a 200GB repository to the Hub. Today, there are a number of ways to do this , but all use a file-centric approach. To bring faster file transfers to the Hub, we've open-sourced xet-core and hf_xet , an integration with huggingface_hub which uses a chunk-based approach written in Rust.
If you consider a 200GB repository with unique chunks, that's 3 million entries (at ~64KB per chunk ) in the content-addressed store (CAS) backing all repositories. If a new version of a model is uploaded or a branch in the repository is created with different data, more unique chunks are added, driving up the entries in the CAS.
With nearly 45PB across 2 million model, dataset, and space repositories on the Hub, a purely chunk-based approach could incur 690 billion chunks . Managing this volume of content using only chunks is simply not viable due to:
Efficient data transfer and storage directly lower infrastructure costs and accelerate model iteration, enabling faster AI development cycles and reducing bottlenecks for large‑scale model sharing.