OSCAR: A Spectral Covariance-Aware Rotation Method for 2-bit KV Cache Quantization

OSCAR (Offline Spectral Covariance-Aware Rotation) addresses long context LLM services' KV cache memory bottleneck via 2-bit quantization. It offline estimates attention-aware covariance structures to derive rotation and cropping thresholds, achieving 8x memory compression, up to 7x throughput improvement, and maintaining BF16-level precision. This work is critical for making long context LLM services economically feasible.

As LLM context windows expand to 128K+ tokens, KV cache memory usage becomes a key deployment bottleneck, limiting batch size and throughput. Quantization reduces memory but 2-bit (INT2) faces two core issues: 1) Simple methods cause sharp precision drops; 2) High-precision methods often require complex custom kernels, hard to integrate into existing frameworks.

OSCAR's core innovation aligns KV quantization with the attention mechanism's covariance structure. Offline steps: 1) Collect Query-Key interaction samples from representative datasets; 2) Estimate covariance patterns from these interactions; 3) Derive rotation matrices that minimize quantization error's impact on attention. This alignment ensures 2-bit quantized KV retains key info for attention computation.

OSCAR provides a deployable system: 1) Custom INT2 attention kernels compatible with paged KV cache (e.g., vLLM), using fusion pipelines for low latency; 2) Seamless integration into mainstream frameworks like vLLM and SGLang, allowing users to benefit without modifying application code.

OSCAR is validated across models: 1) Small/medium models (Qwen3-4B/8B): OSCAR's precision gap vs BF16 is only 3.78/1.42 percentage points, while naive INT2 rotation fails; 2) Large models (32B, 358B): Maintains BF16-level precision; 3) Long context (128K RULER-NIAH): OSCAR remains stable, naive INT2 fails.

System gains: 8x KV cache memory reduction, up to 7x throughput (large batches), up to 3x decoding speed (memory bandwidth optimization). Conclusion: OSCAR solves 2-bit KV quantization's precision problem, enabling long context LLM services economically and driving their broader application.