IceCache: A New Efficient KV Cache Management Scheme for Long-Sequence Large Language Models

In long-sequence inference tasks, KV cache management for Large Language Models (LLMs) is a key bottleneck for performance and resource efficiency. IceCache innovatively combines semantic token clustering and paged attention mechanisms to achieve 99% of the original model accuracy while retaining only 25% of the cache budget, providing a memory-efficient solution for long-sequence inference.

KV cache stores intermediate attention states during autoregressive generation to accelerate inference, but the cache size is proportional to the sequence length. In long-sequence scenarios, this easily leads to GPU memory exhaustion or slowdowns due to data transfer. Traditional token selection strategies (such as retaining recent tokens) are simplistic and prone to losing key semantic information.

The core idea of IceCache is to organize semantically related tokens into contiguous memory regions and manage them dynamically:

1. Semantic Clustering: Retain semantically representative tokens instead of selecting only by position, reducing cache usage while maintaining performance;
2. Paged Attention: Optimize memory layout and access patterns to efficiently utilize bandwidth and reduce CPU-GPU data transfer overhead.

In LongBench benchmark tests:

• Maintains 99% original accuracy with a 256-token budget;
• With only 25% of the KV cache token budget, latency and accuracy are better than or comparable to offloading-based methods, supporting longer sequences or more concurrent requests.

IceCache uses a hierarchical dynamic data structure that supports real-time adjustment of cache content during generation, adapting to variable-length sequences and streaming inputs; it is compatible with existing inference frameworks and provides open-source implementations and documentation (available on the project website).

IceCache is of great significance for LLM inference optimization:

• Supports long-document RAG systems, long-conversation chatbots, and multi-step reasoning tasks;
• Its memory efficiency advantage makes it possible to run long-sequence inference on consumer GPUs/mobile devices, promoting the popularization of LLM applications.

IceCache combines semantic understanding with system optimization to break through the limitations of traditional methods and balance memory efficiency and model performance. As long-sequence applications grow, such innovative solutions will play a key role in LLM infrastructure.