SparseX: Efficient Segment-Level KV Cache Sharing for Interleaved LLM Services

This paper proposes SparseX, an efficient segment-level KV cache sharing method for long-context LLM services. Addressing the problem that traditional Prefix Cache cannot handle non-prefix segment repetition across requests, rounds, and agents, SparseX restores cross-segment context interactions through segment-level cache reuse, sparse Q indexing to estimate key tokens, and sparse recomputation in a single forward pass. It is compatible with vLLM/PagedAttention and suitable for scenarios like multi-turn dialogue and RAG.

KV cache is the core of LLM inference acceleration. vLLM's Prefix Cache can reuse identical prompt prefixes, but in real-world scenarios, repeated content often appears as non-continuous, interleaved segments (e.g., multi-turn dialogue history, document fragments in RAG, shared context among agents), which traditional mechanisms cannot effectively capture.

• Segments as Reuse Units: Use continuous token segments as basic units, maintain a segment cache pool, and flexibly reuse repeated segments at any position.
• Sparse Q Indexing: Identify key tokens (pronouns, conjunctions, etc.) that require cross-segment context through attention weight distribution.
• Sparse Recomputation in Single Forward Pass: No model modification needed; complete KV recomputation for key tokens in a single forward pass, avoiding extra overhead and maintaining a unified execution path.

• Layer-Specific Hybrid Attention: Keep full attention in early layers (to extract basic features) and switch to sparse recomputation in later layers (for abstract semantic integration), balancing efficiency and quality.
• vLLM Compatibility: Fully supports PagedAttention, Prefix Cache, and FlashAttention backends; model-agnostic, allowing existing vLLM users to upgrade seamlessly.

Suitable Scenarios: Multi-turn dialogue systems, Retrieval-Augmented Generation (RAG), agent workflows, long document processing. Performance Expectations: Significantly reduce prefill latency and computational costs, especially in scenarios with high cache hit rates.

• Expand cache reuse scope: from prefix-level to segment-level.
• Propose sparse recomputation paradigm: selectively recompute key tokens.
• Training-agnostic optimization: deployable without fine-tuning.
• Ecosystem compatibility: deep integration with vLLM, lowering adoption barriers.

• Limitations: The accuracy of key token estimation depends on attention analysis; performance in extremely long contexts (1M+ tokens) remains to be verified.
• Future Directions: Improve the reliability of key token estimation, support multimodal expansion, dynamically adjust layer thresholds.

SparseX breaks through the limitations of traditional Prefix Cache through segment-level KV cache sharing and sparse recomputation, handles complex interleaved repetition patterns, is compatible with existing systems, and provides an efficient and practical solution for long-context LLM services. It is an innovative training-agnostic inference optimization paradigm.