DefensiveKV: Addressing the Vulnerability of KV Cache Eviction in LLM Inference

DefensiveKV is the official implementation of an ICLR 2026 paper. It proposes a systematic solution to the vulnerability issue of KV cache eviction strategies in large language model (LLM) inference, significantly improving the stability of long-context reasoning. This thread will introduce its background, methods, experimental results, and application value in separate floors.

In LLM autoregressive generation, KV cache reduces the computational complexity of attention from quadratic to linear by caching key-value vectors of previous tokens, thus improving inference efficiency. However, as the context length increases, linear growth in memory usage becomes a bottleneck. Existing eviction strategies (such as retaining recent/high-attention tokens) have vulnerabilities that may lead to a sudden drop in generation quality or even crashes, as they ignore the temporal dynamics of attention patterns and inter-layer dependencies.

The core contributions of DefensiveKV are: 1. Establishing a vulnerability analysis framework to quantify the risk of eviction strategies; 2. Proposing a defensive eviction mechanism that evaluates the impact of eviction on future generation and maintains risk scores; 3. Implementing multi-level risk modeling (token/layer/head level), dynamic budget allocation (adjusting cache quota based on task complexity), and fallback recovery mechanism (reloading key tokens when quality degradation is detected).

In long-context benchmark tests, DefensiveKV outperforms methods like H2O and StreamingLLM in generation quality under the same cache constraints, especially in long-distance dependency tasks. More importantly, it improves inference stability: traditional strategies tend to crash under adversarial inputs or edge cases, while DefensiveKV remains stable, making it suitable for production environment deployment.

DefensiveKV is applicable to: 1. Long document processing (summarization, Q&A, code analysis), handling tens of thousands of tokens with limited GPU memory; 2. Multi-turn dialogue systems, intelligently retaining key historical information to maintain coherence; 3. Real-time streaming generation (voice assistants, translation), dynamically balancing latency and quality.

The open-source DefensiveKV by FFY0 is integrated with HuggingFace Transformers, supporting models like Llama, GPT-NeoX, and Mistral. Developers can enable it via a simple API. Limitations include: the computational overhead of defensive eviction needs optimization; the risk assessment model is heuristic-based, and learning-based methods can be explored in the future.

DefensiveKV brings theoretical insights and practical solutions to KV cache management, solving the eviction vulnerability problem and laying the foundation for more reliable and efficient long-context reasoning systems. As LLM applications expand, such innovations will enhance user experience and reduce deployment costs.