Introducing AutoRound: Intel’s Advanced Quantization for LLMs and VLMs

As large language models (LLMs) and vision-language models (VLMs) continue to grow in size and complexity, deploying them efficiently becomes increasingly challenging. Quantization offers a solution by reducing model size and inference latency. Intel's AutoRound emerges as a cutting-edge quantization tool that balances accuracy, efficiency, and compatibility.

AutoRound is a weight-only post-training quantization (PTQ) method developed by Intel. It uses signed gradient descent to jointly optimize weight rounding and clipping ranges, enabling accurate low-bit quantization (e.g., INT2 - INT8) with minimal accuracy loss in most scenarios. For example, at INT2, it outperforms popular baselines by up to 2.1x higher in relative accuracy . The image below provides an overview of the core algorithm in AutoRound. For more details, please refer to our paper .

Despite its strong performance, AutoRound is fast and lightweight — quantizing a 72B model takes just 37 minutes on an A100 GPU under light mode. It also supports mixed-bit tuning, lm-head quantization, GPTQ/AWQ/GGUF format exporting, and flexible tuning recipes.

Most GPTQ/AWQ models can be converted to the AutoRound format for better compatibility and support with Intel devices. Please note that the quantization config will be changed if the model is serialized.

AutoRound delivers highly promising results, particularly in low-bit quantization scenarios. Evaluations across a variety of tasks show that it outperforms popular methods by a wide margin at 2-bit precision (source) . At 4 bits, AutoRound continues to hold a competitive edge in most cases, as demonstrated on the Low-Bit Open LLM Leaderboard .

LLMs: AutoRound supports nearly all popular LLM architectures, including well-known models like Qwen, LLaMA, and DeepSeek. Ready-to-use quantized models are available on Hugging Face through collections such as OPEA , Kaitchup , and fbaldassarri .

VLMs: AutoRound supports over 10 vision-language models (VLMs), including Mistral-Small-3.1, Gemma3, and more. You can find the full list in the README , and ready-to-use quantized models are available in the OPEA Hugging Face collection . For models not yet supported, you can still apply our RTN method with --iters 0 . No tuning is required, but some accuracy loss is expected.