VaSE: Value-Aware Stochastic KV Cache Eviction Strategy for Reasoning Models

VaSE addresses the KV cache memory bottleneck caused by long-sequence outputs of reasoning models by proposing a value-aware stochastic KV cache eviction strategy. This strategy maintains reasoning coherence by protecting large-value states and increases cache diversity by introducing randomness. Under 4x KV cache compression, the reasoning model's average accuracy across six reasoning tasks surpasses SOTA selection methods, outperforming the strongest eviction method by over 4%, and it can be deployed without training.

Reasoning models improve accuracy through chain-of-thought, but long outputs lead to huge KV cache memory usage. Existing KV cache eviction methods reduce costs, but their performance is usually inferior to sparse attention schemes that retain full cache. How to compress KV cache while maintaining model performance is a key challenge currently.

VaSE consists of two core components:

1. Value-aware component: Identify and protect large-value states; retain the top 5-10% of large-value states by setting thresholds to ensure key reasoning clues are not evicted;
2. Stochastic component: Use Gumbel sampling to randomly select from evictable candidates with probability inversely proportional to importance, increasing cache diversity. This method requires no training and acts as an attention mechanism wrapper layer to dynamically decide which KV pairs to retain.

Experiments show that the Qwen3 model using VaSE achieves higher average accuracy across six reasoning tasks than SOTA selection methods with the same sparsity under 4x KV cache compression, outperforming the strongest eviction method by over 4%. Additionally, VaSE supports FlashAttention2 and can achieve static memory usage, which is crucial for production deployment.

VaSE has significant practical value: it can be applied immediately to any Transformer reasoning model without model retraining or architecture modification; the guarantee of static memory usage allows system administrators to accurately predict memory requirements and avoid OOM errors caused by input length changes.

The paper proposes future research directions:

• Dynamic threshold adjustment: automatically determine the protection ratio based on input characteristics;
• Combining with quantization techniques to further compress cache size;
• Adaptive eviction strategy for multi-task scenarios.