Quantized Large Language Models: A Systematic Study of Confidence Calibration

This article interprets the uncertainty-aware-inference project, systematically analyzes the impact of Post-Training Quantization (PTQ) on the confidence calibration of large language models (LLMs) of different scales, and finds that quantization impairs calibration quality (the lower the precision, the larger the model scale, and the more obvious the impact on generation tasks). It also verifies that knowledge distillation can effectively restore part of the calibration performance, and provides practical insights such as quantization strategy selection, post-calibration processing techniques, and monitoring evaluation.

Large language models have high deployment costs, and Post-Training Quantization (PTQ) technology is widely used in resource-constrained environments (compressing weights to low precision such as 8-bit or 4-bit). However, whether quantization affects model reliability—especially the confidence calibration ability (the ability of prediction probability to reflect actual correctness)—is a key issue. The uncertainty-aware-inference project conducts a systematic study on this.

Confidence calibration refers to the matching between the model's prediction confidence and actual accuracy. Poor calibration (overconfidence/underconfidence) affects decision reliability. In high-risk scenarios such as medical diagnosis (affecting doctors' adoption of AI recommendations), autonomous driving (determining the timing of human takeover), and financial risk control (affecting the rate of misjudgment and missed judgment), calibration quality is crucial.

Model Coverage: Covers architectures such as LLaMA, Mistral, Falcon, parameter scales from 7B to 70B, and quantization precisions like INT8/INT4; Evaluation Metrics: Uses standard metrics such as ECE (Expected Calibration Error), MCE (Maximum Calibration Error), reliability diagrams, and Brier score to measure calibration quality.

Negative Impact of Quantization: PTQ impairs calibration quality; the lower the precision (e.g., INT4 vs. INT8), the larger the model scale, and the more obvious the impact on generation tasks (compared to classification tasks); Recovery via Distillation: Knowledge distillation with a full-precision model as the teacher and a quantized model as the student can significantly improve ECE and partially restore performance, but it requires a trade-off with additional computing resources.

Quantization Strategies: Prioritize INT8, keep key layers at high precision, mixed-precision quantization; Post-Calibration Processing: Temperature scaling, Platt scaling, bucket calibration; Monitoring and Evaluation: Regular sampling to evaluate ECE, establish a baseline for confidence distribution, analyze high-confidence wrong predictions.

The study only focuses on PTQ (commonly used in industry, low cost) and does not involve Quantization-Aware Training (QAT); the evaluation uses multi-task benchmark datasets (question answering, reasoning, code generation, etc.) to enhance the robustness of the conclusions.

Future Directions: Dynamic quantization, calibration-aware quantization objectives, other uncertainty representations; Summary: This study quantifies the impact of PTQ on LLM calibration, verifies the effectiveness of distillation, provides empirical references for deployment, and helps establish more reliable quantized LLM practices.