Fusing Dequantization and GEMM: Practical CUDA Kernel Optimization for LLM Inference

Introduces the fused-dequant-gemm project, whose core is merging dequantization (from INT8 weight quantization) with GEMM via CUDA kernel fusion technology to address the memory bandwidth bottleneck in LLM inference, reduce DRAM consumption, and improve performance. The project was open-sourced by zhangtina0103 and released on GitHub on June 6, 2026.

LLM inference performance is limited by DRAM bandwidth (especially for small batch sizes). INT8 weight quantization compresses weights by 4x, but traditional separate operations (dequantization to FP32 followed by GEMM) introduce unnecessary memory round trips, offsetting the gains.

The project embeds dequantization into the GEMM tile loading process via kernel fusion. Traditional path: INT8 → FP32 buffer → GEMM; Fused path: INT8 → on-the-fly dequantization in registers/shared memory → GEMM, eliminating intermediate buffer reads/writes and reducing memory traffic by approximately 33%.

Uses grouped symmetric INT8 quantization (each group of 128 elements shares a scaling factor). Three CUDA kernels are implemented: independent dequantization, tiled GEMM, and fused kernel. The fused kernel performs on-the-fly dequantization during the loading phase, avoiding writes back to global memory.

Benchmark results: The fused CUDA implementation achieves an effective GFLOPS of 7100, memory bandwidth utilization of 198.4 GB/s, latency of 2.40 ms, and a speedup of 3.38×; compared to the separate implementation, it improves performance by 15% and reduces DRAM traffic by 33%.

Kernel Implementation	Effective GFLOPS	Memory Bandwidth Utilization	Latency	Speedup
Unfused Python	2,100	45.2 GB/s	8.12 ms	1.00×
Unfused cuBLAS	8,400	180.5 GB/s	2.03 ms	4.00×
Separate CUDA	6,200	133.1 GB/s	2.75 ms	2.95×
Fused CUDA	7,100	198.4 GB/s	2.40 ms	3.38×

Nsight monitoring metrics show that after fusion, global memory load traffic is reduced and there are no global storage bytes. Key insights: Weight quantization is most effective for small batches; fusion eliminates intermediate tensor memory traffic; it aligns with the TensorRT-LLM quantization path.

Applicable scenarios: Edge devices, high-throughput services, quantization research. Extension directions: FP8 KV cache compression, cuBLASLt INT8 GEMM benchmarking, vectorized tile loading.

The project demonstrates the key paradigm of kernel fusion to eliminate memory round trips, with significant performance improvements. It provides a learning case for LLM inference optimization engineers, revealing quantization pitfalls and their solutions. Memory bandwidth optimization techniques are becoming increasingly important in the era of large models.