Bringing Nunchaku 4-bit Diffusion Inference to Diffusers

Hugging Face has introduced support for Nunchaku 4-bit diffusion inference directly within Diffusers. Traditionally, loading modern text-to-image models in BF16 precision requires between 20 and 30 gigabytes of VRAM, limiting their use on standard consumer hardware. Quantization solves this, but most existing backends are weight-only and do not accelerate inference speeds. The SVDQuant method, utilized by the Nunchaku inference engine, runs main transformer layers with 4-bit weights and activations, reducing memory usage while accelerating the denoising loop. Through the new Nunchaku Lite integration, users can load Nunchaku-style checkpoints directly using standard loading commands without requiring a separate inference engine or local CUDA compilation. The CUDA kernels are downloaded automatically from the Hub via the kernels package. According to Hugging Face, the approach reduces peak VRAM usage by up to 50 percent while improving latency by approximately 30 percent. Additional optimizations like torch.compile can further enhance end-to-end speedups. Furthermore, the companion diffuse-compressor toolkit allows developers to calibrate, quantize, package, and publish their own model architectures as regular Hugging Face repositories. While Nunchaku Lite does not match the exact speed of the original architecture-specific fused Nunchaku engine due to a lack of custom fused modules, it maintains the same level of VRAM reduction while offering a flexible, architecture-agnostic implementation path for the broader ecosystem.