Deep Understanding of Neural Network Quantization: A Study on Precision and Performance Trade-offs from FP32 to INT8/INT4

This research is a 50-day in-depth project focusing on the quantization process of neural networks from FP32 to INT8/INT4. By implementing custom CUDA kernels and comparing with NVIDIA TensorRT, it explores the trade-off between precision loss and performance improvement, providing practical references for TensorRT and inference teams. The study covers real models such as ResNet-18 and DistilBERT, including systematic benchmarking and layer-wise sensitivity analysis.

While deep learning models have improved in performance, their computational and storage costs have increased dramatically (e.g., FP32 models require several gigabytes of memory), making deployment on edge devices or real-time scenarios difficult. Quantization technology converts high-precision floating-point numbers to low-precision integers (INT8/INT4), which can significantly reduce model size and accelerate inference, but the reduced precision leads to performance degradation. This project aims to understand the extent of this degradation and how to balance it properly.

Core research questions: 1. Precision loss when quantizing FP32 to INT8/INT4; 2. Performance differences in latency and throughput between custom CUDA kernels and TensorRT. The project framework includes: custom CUDA C++ kernels (INT8 quantization/dequantization/GEMM), Python experimental framework (applied to real models), full-precision comparison benchmarking, layer-wise sensitivity analysis, and TensorRT comparison verification.

Hardware Environment: NVIDIA RTX4060 Laptop GPU (Ada Lovelace architecture, 8GB GDDR6, 4th-gen Tensor Core). Software Stack: CUDA Toolkit 12.x, PyTorch 2.x (CUDA12.1), TensorRT 8.6+, CMake3.20+, Nsight tools. Quantization Strategies: Post-Training Quantization (PTQ), Quantization-Aware Training (QAT), Mixed Precision (automatic allocation of INT8/FP16).

• Quantization: The linear quantization formula is quantized_value = round((floating_point_value - zero_point)/scale_factor), where scale factor and zero point are determined by the calibration dataset. - INT8 vs INT4: INT8 (-128127) reduces storage by 4x with controllable precision loss; INT4 (-87) reduces storage by 8x but has more obvious precision loss, requiring fine calibration. - Tensor Core Acceleration: NVIDIA Tensor Core optimizes matrix operations and supports mixed precision; INT8 Tensor Core can reduce quantization overhead at high throughput.

Benchmark Process: Load ResNet-18/DistilBERT → Validate with CIFAR10/ImageNet subsets → Evaluate Top1/Top5 accuracy → Measure latency and throughput → Compare model sizes. Sensitivity Analysis: Input layers/shallow layers are sensitive to quantization (basic feature extraction); deep layers are more tolerant (high-level semantic features); Transformer attention layers require special attention. TensorRT Comparison: Verify the correctness of custom implementations and understand the design trade-offs of industrial-grade solutions.

Significance: Quantization is a core technology of TensorRT; researching underlying principles (scale factor, zero point, trade-offs) is crucial for NVIDIA's inference team. It provides developers with reproducible frameworks, performance tuning guides, and problem diagnosis tools. Application Scenarios: Edge device deployment, real-time inference, cloud optimization. Conclusions: INT8 quantization can retain over 95% of original performance; calibration strategies significantly reduce errors; layer-wise analysis guides mixed precision; hardware co-design improves kernel efficiency.