IndexCache: Accelerating DeepSeek Sparse Attention Inference via Cross-Layer Index Reuse

IndexCache is an inference acceleration technique for DeepSeek's sparse attention model. Its core lies in reusing index calculation results across layers, which significantly reduces computational overhead and improves inference speed while maintaining model quality. This technology leverages the similarity of attention patterns between adjacent layers in the multi-layer Transformer structure, enabling efficient reuse through an intelligent caching strategy. It is suitable for scenarios such as long document processing and real-time dialogue, providing a new direction for the efficient deployment of large models.

The computational complexity of fully connected attention in large language models grows quadratically with sequence length, which is a bottleneck for long text processing. Sparse attention reduces this complexity to linear, but in multi-layer Transformer structures like DeepSeek, there is significant redundancy when each layer computes indices independently. IndexCache addresses this issue by proposing a cross-layer index reuse optimization strategy.

IndexCache is based on the observation that attention patterns of adjacent layers are highly similar. It reuses indices via a caching mechanism: caching indices from one layer for use in subsequent layers, supporting reuse across adjacent layers, different layers of the same sequence, and cross-sample reuse in batches. It also uses an intelligent strategy to balance speed and accuracy: fully reusing indices for similar layers, performing partial updates or recalculations for layers with large changes, and controlling quality loss through thresholds.

IndexCache is tightly integrated with DeepSeek: it intercepts the sparse attention computation process, checks cache availability → evaluates applicability → decides to reuse or recalculate → updates the cache. Memory management uses compact structures, LRU eviction, and memory-VRAM tiering; batch processing optimizations include shared caching for similar sequences, sequence length alignment, and asynchronous prefetching.

IndexCache significantly accelerates inference with minimal quality loss: it provides a 20%-40% speedup in typical long text scenarios while maintaining over 95% output quality. The benefits depend on the number of model layers, sequence length, and stability of attention patterns. Application scenarios include long document processing, real-time dialogue, code generation and understanding, and batch processing tasks.

Limitations of IndexCache: 1. It is designed for DeepSeek's sparse architecture; other models need validation. 2. Index reuse is approximate, so caution is needed in high-precision scenarios (medical, legal). 3. Additional memory overhead may occupy VRAM. 4. Cache hit rate is low for dynamic content input (creative writing).

IndexCache represents an optimization direction of redundancy mining at the algorithm level. It does not require modifying model weights, is a lossless optimization, and can be used in combination with speculative sampling. Future directions include adaptive caching strategies, multi-model support, hardware co-optimization, and cross-device index sharing in distributed scenarios.