FADE: Attention-Aware Hierarchical KV Cache Compression for LLM Inference

FADE (Frequency-Adaptive Decay Encoding) is an attention-aware hierarchical KV cache compression technique for LLM inference. By differentially handling the storage precision of different tokens, it achieves a 3-8x KV cache compression ratio while maintaining near-baseline output quality, effectively addressing the memory bottleneck in long-context inference. Its core innovations lie in the hierarchical cache architecture and flexible eviction strategies, which adapt to various application scenarios.

The inference efficiency of Large Language Models (LLMs) is limited by the memory footprint of KV caches. As context length increases, KV cache grows linearly, becoming the main bottleneck for long-sequence inference. Traditional quantization methods use a uniform compression strategy, ignoring the differential importance of different tokens in the attention mechanism, making it difficult to balance compression ratio and output quality.

The core of FADE is a three-tier dynamic cache architecture:

1. FP16 Full-Precision Layer: Retains anchor tokens (e.g., system instructions) and recent tokens to ensure the integrity of key information;
2. INT4 Quantization Layer: Intermediate tokens are stored with 4-bit quantization, which is the main source of memory savings;
3. INT2 Deep Compression Layer (Optional): Some tokens are further compressed to 2-bit, suitable for scenarios with low quality sensitivity. Eviction strategies include four types: H2O (optimal quality), EMA (streaming generation), Position (simplest), and Learning Strategy (intelligent), adapting to different scenario requirements.

FADE provides three preset configurations:

• Safe Mode: 3-4x compression ratio, 100% greedy decoding matching rate, no eviction;
• Balanced Mode: ~5x compression ratio, uses H2O strategy to balance compression and quality;
• Aggressive Mode:7-8x compression ratio, effect needs verification. It supports mainstream model series (Qwen2/Qwen3, Llama, Mistral, etc.) and multiple RoPE types. A known limitation is that Qwen3.5/3.6 can only compress 25% of full attention layers (DeltaNet layers are not supported).

FADE's effectiveness has been verified on multiple models:

• Qwen2.5-3B-Instruct: Baseline 12.2MiB → Hierarchical 4.0MiB (-67%), 100% greedy decoding matching rate;
• Llama-3.2-1B: Baseline29.9MiB → Hierarchical6.3MiB (-79%), output coherence maintained, eviction rate ~29%. The results show that FADE can significantly reduce memory footprint while maintaining high-quality output.

Advanced features include:

• Session Persistence: Save/restore compressed cache;
• Telemetry Debugging: Export layer allocation events and debug snapshots;
• Product Quantization (PQ): Replace INT2 to achieve ~2bit/element compression. Usage Notes: Only H2O prefill requires eager mode; it is recommended to use auto to select the attention implementation; verify Transformers version (4.45/5.3); use compressed_storage_bytes() to count KV memory; start testing with batch_size=1.

FADE's core contributions: Differentiated token storage, flexible eviction strategies, wide model compatibility, and production-ready configurations. In the future, it is expected to be deeply integrated with inference engines such as vLLM and SGLang to further improve the deployment efficiency of LLM long-context inference and provide practical solutions for memory optimization.