Inference Model Quantization Practice: A Complete Experimental Path from 8-bit Baseline to 4-bit Recovery

This study systematically explores the complete experimental process of Transformer inference model quantization, covering the establishment of an 8-bit baseline, performance degradation caused by aggressive 4-bit quantization, and strategies for performance recovery using QLoRA and GRPO. The study validates the impact of quantization on inference capabilities on the GSM8K (mathematical reasoning) and GPQA (scientific question answering) benchmarks and provides a reproducible code framework.

With the improvement of inference capabilities of large language models, model size and resource requirements have grown exponentially. Quantization, as a model compression method, can reduce memory usage and latency, but inference tasks are more sensitive to precision than text generation—aggressive quantization easily leads to significant performance degradation. This study explores the application of Post-Training Quantization (PTQ) in Transformer inference models, focusing on the performance change curve from 8-bit to 4-bit quantization and fine-tuning recovery strategies, and selects GSM8K and GPQA to evaluate the impact of quantization.

A phased experimental design is adopted, with the tech stack based on PyTorch and the Hugging Face Transformers ecosystem, core dependencies including bitsandbytes (low-precision quantization), PEFT (parameter-efficient fine-tuning), and TRL (reinforcement learning). The experimental environment uses an NVIDIA H100 NVL GPU (93GB memory), CUDA 12.8, and Python 3.13.2, while exploring the feasibility in resource-constrained environments. The project centers on a Jupyter Notebook (DL23.ipynb), with the code structure divided into three layers: source code (src/), scripts (scripts/), and configurations (configs/).

8-bit quantization (INT8) reduces model memory usage by approximately 50% while retaining most of the original precision. In benchmark tests on GSM8K and GPQA, the accuracy drop is controlled within an acceptable range, providing a reference for subsequent aggressive quantization and verifying the feasibility of 8-bit quantization in practical deployment.

4-bit quantization (INT4) compresses model weights to 1/4 of the original size, significantly reducing memory requirements, but leads to a notable drop in inference capabilities: the mathematical reasoning accuracy on GSM8K decreases obviously, the integrity of multi-step reasoning chains is broken; the performance of GPQA scientific question answering tasks degrades, and complex logical deductions are prone to errors. The reason is the cumulative effect of quantization errors—discretization errors gradually amplify during the reasoning process, leading to incorrect conclusions.

To address 4-bit quantization losses, the QLoRA technique is introduced: freeze the 4-bit quantized weights, learn to compensate for quantization errors through low-rank adapters, and achieve parameter-efficient fine-tuning. After completing the training process, the QLoRA adapter fine-tuned on GSM8K and GPQA data significantly improves inference capabilities and partially recovers quantization losses. QLoRA only requires training less than 1% of the adapter parameters, achieving results close to full-precision fine-tuning, making it suitable for resource-constrained environments.

During decoding, strategies such as temperature adjustment, sampling optimization, and chain-of-thought prompting are tested to mitigate the impact of quantization; the GRPO reinforcement learning framework is introduced to optimize reasoning behavior through reward shaping. Comparisons show that the QLoRA+GRPO combination further improves performance on complex reasoning tasks compared to pure QLoRA, and reinforcement learning feedback helps the model avoid reasoning paths with cumulative quantization errors.

Practical recommendations: Prioritize 8-bit quantization in scenarios with sufficient memory and critical tasks; try the 4-bit+QLoRA combination with fine-tuning for edge deployment or high-throughput scenarios. Future directions: Explore mixed-precision quantization (different bit widths for attention layers and FFN layers), combination of activation quantization and weight quantization, and adaptive quantization methods for specific inference tasks.