Migrating the Hub from Git LFS to Xet

In January of this year, Hugging Face's Xet Team deployed a new storage backend, and shortly after shifted ~6% of Hub downloads through the infrastructure . This represented a significant milestone, but it was just the beginning. In 6 months, 500,000 repositories holding 20 PB joined the move to Xet as the Hub outgrows Git LFS and transitions to a storage system that scales with the workloads of AI builders.

Today, more than 1 million people on the Hub are using Xet. In May, it became the default on the Hub for new users and organizations . With only a few dozen GitHub issues, forum threads, and Discord messages, this is perhaps the quietest migration of this magnitude.

How? For one, the team came prepared with years of experience building and supporting the content addressed store (CAS) and Rust client that provide the system's foundation. Without these pieces, Git LFS may still be the future on the Hub. However, the unsung heroes of this migration are:

Driven by our commitment to the community, these seemingly straightforward decisions had significant implications. Most importantly, we did not believe users and teams should have to immediately alter their workflow or download a new client to interact with Xet-enabled repositories.

Instead of waiting for all upload paths to become Xet-aware, forcing a hard cut-over, or pushing the community to adopt a specific workflow, we could begin migrating the Hub to Xet immediately with minimal user impact. In short, let teams keep their workflows and organically transition to Xet with infrastructure supporting the long-term goal of a unified storage system.

If you upload or download from the Hub using your browser or use Git, that's fine. Chunk-based support for both is coming soon. In the meantime use whichever workflow you already have; no restrictions.

For older versions of huggingface_hub or huggingface.js , which do not support chunk-based file transfers, you can still download and upload to Xet repos, but these bytes take a different route. When a Xet-backed file is requested from the Hub along the resolve endpoint, the Git LFS Bridge constructs and returns a single presigned URL , mimicking the LFS protocol. The Bridge then does the work of reconstructing the file from the content held in S3 and returns it to the requester.

We now support some of the largest creators on the Hub ( Meta Llama , Google , OpenAI , and Qwen ) while the community keeps working uninterrupted.

Together, these components have allowed us to aggressively migrate PBs in the span of days without worrying about the impact to the Hub or the community. They're giving us the peace of mind to move even faster in the coming weeks and months ( skip to the end 👇 to see what's coming).

In the early days of planning the migration to Xet, we made a few key design decisions:

If you have a Xet-aware client (e.g., hf-xet , the Xet integration with huggingface_hub ), uploads and downloads pass through the entire Xet stack. The client either breaks up files into chunks using content defined chunking while uploading, or requests file reconstruction information when downloading. On upload, chunks are passed to CAS and stored in S3 . During downloads, CAS provides the chunk ranges the client needs to request from S3 to reconstruct the file locally.