ReSET: An Inference Step-Aware Temperature Scaling Method for NVFP4 Quantization

ReSET is an inference step-aware temperature scaling method for NVFP4 quantization proposed by the AIHA Lab team. It aims to solve the accuracy loss problem of NVFP4 quantization in inference models, while achieving significant speedup through CUDA kernel optimization. This method was released on arXiv on June 11, 2026, and the open-source code is available at https://github.com/aiha-lab/ReSET. Key highlights include: online estimation of inference step-level uncertainty and adaptive adjustment of decoding temperature, as well as CUDA kernel optimization for small-batch autoregressive decoding (achieving a 2.5x speedup compared to NVFP4 vLLM implementation).

Large Reasoning Models (LRMs) enhance the ability to solve complex problems, but their inference cost is high (more token generation, large memory usage). As a hardware-supported low-precision scheme, NVFP4 can reduce computation and memory costs, but its direct application to inference models has two major limitations: 1. Quantization leads to a decrease in inference accuracy; 2. Existing NVFP4 kernels do not fully leverage their latency advantages in small-batch autoregressive decoding scenarios. Further analysis shows that the impact of quantization on the inference process includes: increased probability of incorrect sampling of low-entropy symbolic tokens (prone to errors at key decision points), and models in high-uncertainty steps being overly focused on a few tokens (losing diversity).

Step-Level Temperature Scaling: Adaptive temperature adjustment based on inference step entropy, combining token-level entropy (uncertainty distribution of candidate tokens in the current step) and step-level entropy (global uncertainty of the entire inference step) to dynamically adjust temperature—increase temperature to enhance diversity when the model is overly confident, and decrease temperature to focus on key options when it is overly dispersed.

CUDA Kernel Optimization: Design a small-M NVFP4 kernel for small-batch autoregressive decoding, solving the small-batch latency bottleneck of traditional NVFP4 implementations through fine-grained thread scheduling and memory access pattern optimization.

Experimental results show:

• Accuracy: On multiple inference benchmarks and different model scales, ReSET improves by about 2 percentage points compared to the NVFP4 baseline, narrowing the performance gap with full-precision inference;
• Efficiency: The CUDA kernel achieves a 2.5x kernel-level speedup compared to NVFP4 vLLM implementation, and end-to-end decoding achieves about 2x speedup compared to the BF16 baseline.

Technical Insights: When applying quantization technology to inference models, the hierarchical structure of uncertainty in the inference process must be considered—simple token-level processing cannot capture this; through algorithm design and low-level optimization, it is possible to minimize the damage to model capabilities while maintaining low-precision efficiency.

Future Directions: Extend to other low-precision formats (such as INT4, FP8), or combine with inference optimization techniques like speculative decoding and early stopping strategies to further improve deployment efficiency.