Hugging Face has introduced Storage Buckets, a mutable, S3‑like object storage layer on the Hub aimed at handling the high‑throughput, non‑versioned artifacts that arise during machine‑learning training. Buckets are built on Xet’s chunk‑based storage backend, which breaks files into chunks and deduplicates them across uploads; this reduces bandwidth, speeds transfers, and lowers storage footprints, especially for Enterprise customers whose billing is based on deduplicated usage. The service can be accessed through the hf CLI, Python huggingface_hub, JavaScript @huggingface/hub, or via the fsspec‑compatible HfFileSystem, allowing developers to list, read, write, and glob bucket contents with standard filesystem operations. Buckets support pre‑warming, letting users bring hot data closer to the compute region to avoid cross‑region latency, and are currently partnered with AWS and GCP, with plans to add more providers. Users can create a bucket, sync local directories, inspect contents, and later promote stable artifacts to versioned model or dataset repos. The feature was beta‑tested with launch partners before being released to all users. Free accounts receive starter storage, while PRO and Enterprise plans offer higher limits and deduplicated billing benefits.
Introducing Storage Buckets on the Hugging Face Hub
Hugging Face has launched Storage Buckets, a mutable, S3‑like object storage layer on the Hub designed for fast, non‑versioned ML artifacts such as checkpoints, logs, and processed data. Buckets are built on Xet’s chunk‑based deduplication, enabling efficient bandwidth and storage use, and can be managed via the hf CLI, Python, JavaScript, or fsspec‑compatible APIs. The service supports pre‑warming to reduce cross‑region latency and offers integration with AWS and GCP. Buckets are free for basic use and included in PRO and Enterprise plans, with deduplicated billing for Enterprise customers.
Why it matters
By offering a dedicated, efficient storage layer for transient ML artifacts, Hugging Face streamlines training pipelines and reduces infrastructure costs, benefiting AI developers and enterprises alike.