TurboQuant: A KV Cache Quantization Scheme Approaching Theoretical Limits, Enabling 2-4 Bit Compression with Lossless Inference Quality

TurboQuant, open-sourced by Aitherium, uses random rotation and Beta distribution quantization techniques to achieve nearly lossless LLM inference quality with 2.5-3.5 bit compression. It effectively solves the KV cache memory bottleneck, bringing breakthrough memory optimization for edge deployment and long-context applications.

In modern LLM inference, KV cache memory usage can exceed the model parameters themselves (e.g., a 70B model requires hundreds of GB of VRAM to process 128K context), limiting long-context processing and edge deployment. Traditional quantization methods often sacrifice model quality, and balancing compression ratio and performance is an industry challenge.

1. Random Rotation and Beta Distribution: Randomly rotate input vectors so that coordinates follow a concentrated Beta distribution and are approximately independent. This allows the application of optimal scalar quantization and is data-independent without requiring specific training. 2. Two-Stage Inner Product Quantization: First use an MSE quantizer, then apply a 1-bit Johnson-Lindenstrauss (JL) transform to the residuals. This solves the inner product estimation bias and maintains the accuracy of attention calculations.

Theoretically, it approaches the information-theoretic lower bound (with a gap of about a 2.7x constant factor). KV cache quantization experiments: 3.5 bits achieve quality neutrality, while 2.5 bits result in negligible quality degradation. In nearest neighbor search, its recall rate is better than product quantization, and the indexing time is almost zero.

Open-sourced based on arXiv paper 2504.19874, with a clear code structure that is easy to integrate. It is plug-and-play, requiring no calibration or training for specific models, helping developers reduce inference costs, expand context capabilities, or enable edge deployment.

• Long-context processing: Support longer context with limited VRAM (document analysis, multi-turn dialogue, etc.).
• Edge deployment: Mobile/edge devices can run larger models.
• Cost optimization: Reduce hardware costs and improve energy efficiency.
• Real-time applications: Low memory bandwidth requirements result in faster inference speeds.

TurboQuant represents an important advancement in the field of KV cache quantization. Its compression efficiency approaching theoretical limits and quality preservation capability open up new possibilities for efficient LLM deployment, making it an open-source project worth the attention and trial of researchers and engineers.