AsymCache: A Computation Latency-Aware KV Cache Management System for LLM Inference

The original author team (arXiv:2606.02964v1) released the AsymCache system on arXiv on June 1, 2026. This system achieves lossless KV cache management through three key innovations: multi-segment attention mechanism, jointly optimized eviction strategy, and adaptive chunk scheduling. Experiments show that AsymCache can reduce TTFT of LLM inference by 1.90-2.03x and TPOT by 1.62-1.71x, and further reduce the average job latency by 18.1% in agent service systems, providing an efficient solution for long-context and complex reasoning scenarios.

KV cache is the performance cornerstone of LLM inference. It avoids repeated attention computation by storing key-value vectors of historical tokens, but its memory usage increases linearly with sequence length, easily becoming a GPU memory bottleneck. Among existing solutions, approximate methods trade accuracy for memory, while lossless methods decide eviction only based on access frequency/location without considering the impact of KV cache blocks on GPU attention kernel efficiency, leading to a disconnect between decisions and computation latency characteristics.

1. Multi-Segment Attention (MSA)：Breaks the traditional assumption of continuous caching, supports efficient processing of non-continuous KV contexts, and provides a foundation for flexible eviction of cache blocks; 2. Jointly Optimized Eviction Strategy：Simultaneously optimizes cache hit rate and location-aware recomputation cost, balancing computation and cache efficiency; 3. Adaptive Chunk Scheduler：Dynamically adjusts processing granularity based on workload and GPU status to maximize hardware utilization.

AsymCache performs excellently on common workloads: TTFT is reduced by 1.90-2.03x (reducing computation overhead in the prefill phase), TPOT by 1.62-1.71x (improving autoregressive generation efficiency); after integration with agent systems (e.g., Continuum), the average job latency is further reduced by 18.1%, verifying its value in complex reasoning scenarios.

Design Insights: KV cache management needs to shift from memory optimization to computation-memory co-optimization; non-continuous KV cache processing is feasible and efficient; adaptive scheduling is crucial for dynamic workloads. Summary: AsymCache achieves computation latency-aware KV cache management through three innovations, providing a new paradigm for LLM inference, especially suitable for long-context and complex reasoning scenarios.

AsymCache technology is applicable to: 1. Long-context inference: Solves the memory bottleneck of KV cache for long sequences; 2. Multi-turn dialogue systems: Supports limited memory management for longer dialogue histories; 3. Agent workflows: Improves the performance of complex agent workflows, as shown in the experiment with Continuum integration.