DASH-KV: Asymmetric KV Cache Hashing Accelerates Long-Context LLM Inference

DASH-KV is an asymmetric KV cache compression method proposed in ACL 2026 Findings. It addresses the memory explosion and computational complexity issues in long-context LLM inference by using asymmetric hashing for Key (K) and Value (V) vectors. This approach maintains model performance while significantly reducing memory and computational overhead, and can be integrated into existing frameworks without retraining.

Long-context processing is critical for LLMs but faces two main bottlenecks: 1) KV cache memory grows linearly with sequence length (e.g., a 7B model handling 100K tokens needs dozens of GB of VRAM), leading to frequent memory swaps; 2) Attention computation complexity is O(n²). Existing solutions include quantization (limited compression ratio, numerical errors), pruning (risk of losing key info), paging/swapping (I/O overhead), and sparse attention (needs retraining).

DASH-KV's key innovation is asymmetric hashing for K and V:

• Key Compression: Uses lightweight Local Sensitive Hashing (LSH) to cluster similar keys, preserving semantic info needed for accurate attention scores.
• Value Compression: Adopts more aggressive strategies (coarse-grained quantization/clustering) since value vectors are weighted averaged (errors are smoothed). This design leverages the insight that attention accuracy depends on K quality, while output robustness tolerates V's moderate compression.

DASH-KV's implementation includes:

1. Dynamic Hash Table: Manages compressed KV cache, updating clusters/codebooks as new tokens are generated.
2. Approximate Attention: Compares queries with hash bucket centers instead of individual keys, reducing computation from linear to sublinear.
3. Adaptive Compression: Adjusts compression rate dynamically (lower at critical positions like document boundaries, higher elsewhere).
4. Framework Integration: Pluggable into mainstream frameworks (vLLM, TensorRT-LLM) without modifying model weights.

Experimental results show:

• Memory Efficiency: Significant compression allows longer contexts on same hardware or same context on cheaper hardware.
• Speed: Reduced memory access and attention computation boost long-sequence inference throughput.
• Model Quality: Minimal impact on performance across long-context tasks (close to original model).
• Scalability: Advantages become more obvious as context length increases.

DASH-KV is suitable for:

• Long Document Processing: Lower hardware requirements for books/reports summarization.
• Multi-turn Dialogue: Maintains dialogue history without slowing response.
• Code Understanding: Handles large codebases on resource-limited devices.
• Edge Deployment: Enables long-context models on consumer GPUs/edge devices.

DASH-KV stands out:

• No Retraining: Directly applies to pre-trained models (lower threshold).
• Full Attention: Preserves complete attention mechanism (no performance loss from architecture changes).
• Dynamic Adaptation: Adjusts to context changes (unlike static compression).
• Fine-grained Control: Allows users to balance efficiency and quality.

DASH-KV provides a promising solution for long-context LLM inference via asymmetric hashing, promoting wider deployment of long-context applications. Future directions:

• Combine quantization and hashing for higher compression.
• Optimize for specific domains (code, legal docs).
• Hardware-aware compression to leverage GPU memory hierarchy.
• Extend asymmetric compression to model parameters.