Hugging Face has introduced Quantization-Aware Healing, a technique designed to recover capabilities in large language models that have undergone structural compression and 4-bit quantization. Standard pipelines typically suffer from degradation in reasoning, mathematics, and coding when reducing parameters and quantizing weights, prompting recovery steps like quantization-aware training or quantization-aware distillation. However, these traditional methods either remain costly and unstable or are limited by the ceiling of a structurally compromised intermediate checkpoint. Quantization-Aware Healing addresses this by distilling directly from the original, uncompressed, full-precision teacher model using a memory-efficient chunked KL-divergence loss for long contexts. According to the source, when applied to a GPT-OSS 120B model compressed to 60B parameters and quantized to MXFP4, the resulting model outperformed its own bfloat16 source checkpoint on seven of nine benchmarks, with notable gains in long-context reasoning and math. Furthermore, the 4-bit model surpassed the full-size teacher on LiveCodeBench and achieved competitive results on GPQA Diamond. When compared directly against quantization-aware training on a 9B model, Quantization-Aware Healing reached peak performance about seven times faster and avoided the sharp performance collapse observed when quantization-aware training ran past its optimal point.
Quantization-Aware Healing: a compressed, 4-bit model that outperforms its full-precision original
Hugging Face researchers have introduced Quantization-Aware Healing, a new recovery method for large language models that have undergone both structural compression and 4-bit quantization. By distilling directly from the original pre-compression teacher model rather than a recovered checkpoint, the technique yields a compressed model that outperforms its full-precision source.
Why it matters
This approach challenges the conventional assumption that compression and quantization inevitably degrade model capabilities, offering a stable and efficient pathway to deploy smaller, cheaper, and more accurate models.