AutoRound: Intel's Open-Source Large Model Quantization Tool for Low-Bit High-Precision Inference

AutoRound is an advanced open-source large language model quantization toolkit by Intel, supporting ultra-low-bit quantization (2-4 bits). It optimizes rounding strategies via signed gradient descent, significantly reducing model storage and inference costs while maintaining high precision. Adopting a post-training quantization (PTQ) paradigm, it requires no original training data or fine-tuning—only a small amount of calibration data to complete quantization. It has also been integrated with mainstream frameworks like vLLM and Transformers, providing an efficient and user-friendly solution for large model deployment.

As the parameter scale of large language models rises from billions to hundreds of billions, storage and inference costs have become major bottlenecks for widespread adoption. Quantization technology, as an important model compression method, can significantly reduce memory usage and accelerate inference by lowering the precision of weights and activations. AutoRound is a quantization solution developed to address this need.

The core innovation of AutoRound lies in using signed gradient descent to optimize rounding decisions for weight quantization, which is superior to traditional nearest-neighbor rounding. Based on the post-training quantization (PTQ) paradigm, it does not require access to original training data or fine-tuning—only 128-512 calibration samples are needed, and quantization of a 7B model can be completed in about 10 minutes, lowering the application threshold.

1. Ultra-low bit precision with high accuracy: Maintains strong performance in 2-3 bit scenarios and leads the industry in 4-bit (e.g., DeepSeek-R1 INT2 mixed quantization retains 97.9% of original precision);
2. Cross-hardware support: Optimized for Intel Xeon CPU, NVIDIA GPU, Intel XPU, and Gaudi HPU;
3. Multi-format export: Supports formats like auto_round, auto_awq, and gguf;
4. AutoScheme automatic mixed precision: Specify the target average bit count to automatically generate the optimal scheme;
5. Multimodal support: Compatible with over 10 vision-language models such as Qwen2.5-VL and LLaVA.

Installation

Installation commands for different hardware platforms:

• CPU/NVIDIA GPU: pip install auto-round
• Intel XPU: First install the PyTorch XPU version, then pip install auto-round
• Intel Gaudi: pip install auto-round-hpu

Quantization and Deployment

• Command line: auto-round --model Qwen/Qwen3-0.6B --scheme W4A16 --output_dir ./tmp_autoround
• Python API: Use the AutoRound class to quantize and save
• Inference: Load the quantized model directly in frameworks like vLLM and SGLang.

AutoRound has been integrated into mainstream frameworks such as Transformers (May 2025), vLLM (May 2025), SGLang (October 2025), and LLM-Compressor (November 2025). It has received recommendations from teams like HuggingFace and LMSYS, and quantized models can be directly deployed in production.

Quantization Time

Quantizing a 7B model on a single GPU takes about 10 minutes by default, with adjustable modes:

• High precision: iters=1000
• Balanced: iters=200 (default)
• Fast: iters=50
• RTN: iters=0 (fastest)

Memory Usage

Quantization overhead is 1.1-1.5 times that of the original BF16 model. Enabling low_gpu_mem_usage can save 20GB of VRAM but increases time by 30%.

The AutoRound team continues to push technical boundaries, recently adding support for MXFP4/NVFP4 and FP8 block-level quantization. Through optimized rounding strategies and cross-platform support, it provides an efficient solution for large model deployment, playing an increasingly important role in AI infrastructure and being a preferred tool for developers to reduce inference costs.