IF4: Adaptive Block Scaling Data Type for Optimized Large Model Quantization

As large language models grow in size, model compression techniques have become increasingly important. 4-bit quantization has gained attention for balancing compression ratio and model quality. NVIDIA's NVFP4 is one of the mainstream solutions, but it has the problem of excessive quantization error when values are close to the block maximum. The MIT team proposes the IF4 adaptive block scaling data type, which solves this issue by intelligently selecting FP4 and INT4 representations, providing a more efficient solution for large model compression.

In model compression, quantization techniques reduce storage and computation costs by lowering parameter precision. 4-bit quantization balances well; NVFP4 has hardware support and excellent practical performance, but it has an uneven error distribution problem: in each block of 16 values, values close to the maximum bear disproportionately high quantization errors, affecting model performance. The root cause lies in NVFP4's block scaling strategy—16 values share a scaling factor, and extreme values reduce the representation accuracy of other values.

The core of IF4 is adaptive format selection: based on the distribution characteristics of each block of 16 values, it dynamically selects FP4 (good at dynamic range) or INT4 (suitable for uniform distribution). It cleverly uses the unused sign bit in the E4M3 format of NVFP4's scaling factor to store format information (0=FP4, 1=INT4), with no additional storage overhead. Additionally, this idea extends to IF3 and IF6 formats, reflecting a general design paradigm.

Experiments verify the effectiveness of IF4 in quantization-aware training and post-training quantization scenarios: in quantization-aware training, IF4 models have significantly reduced training loss, can represent parameters more accurately, and capture subtle language patterns; in post-training quantization scenarios, they achieve higher accuracy in downstream tasks such as question answering, text classification, and reasoning, without the need for retraining and with low computational cost.

The hardware feasibility of IF4 is verified through an IF4-supported multiply-accumulate (MAC) unit: this unit efficiently handles FP4 and INT4 operations, with an ingenious circuit design and acceptable area and power consumption overhead. If supported by hardware vendors, IF4 is expected to become the standard quantization format for next-generation AI accelerators, improving representation accuracy at the same bit width and reducing computation and storage costs.

Comparison of IF4 with other quantization methods: compared to 8-bit quantization, it achieves similar model quality with lower storage overhead; compared to 2/3-bit quantization, it provides more guaranteed model quality; compared to complex adaptive methods, its block-level adaptive strategy balances effectiveness and implementability, with better hardware friendliness.

The implementation code of IF4 has been open-sourced (GitHub repository: https://github.com/mit-han-lab/fouroversix) to promote technical application. For large model service providers, IF4 can reduce inference costs and improve response speed; for hardware vendors, supporting IF4 can provide more efficient inference capabilities and form a competitive advantage.

IF4 solves the quantization error problem of NVFP4 near maximum values by adaptively selecting floating-point and integer representations, reflecting a deep understanding of the nature of quantization errors. Combined with hardware feasibility demonstration, IF4 is expected to become an important progress in the field of large model quantization. We look forward to its application and verification in more practical scenarios after the open-source code is released. Paper link: http://arxiv.org/abs/2603.28765v1