ScoutAttention: An LLM Inference Acceleration Scheme for Efficient KV Cache Offloading via Pre-Layer CPU Precomputation

ScoutAttention is a KV cache offloading framework for LLM long-context inference. Through GPU-CPU collaborative block-level sparse attention mechanism and pre-layer CPU precomputation algorithm, it achieves a 2.1x speedup compared to existing offloading methods with an accuracy loss of only 2.4%, effectively solving the GPU memory bottleneck problem in long-context inference.

With the expansion of LLM application scenarios, handling ultra-long contexts has become an essential requirement. However, KV cache memory usage grows linearly. In the Transformer architecture, the memory complexity of KV cache is O(N×H×d). When the sequence length reaches tens of thousands of tokens, it may occupy dozens of gigabytes of GPU memory, limiting batch size and affecting inference throughput and latency.

Existing schemes for offloading KV cache to CPU have two major problems: frequent data transfer overhead leads to PCIe bandwidth bottleneck, leaving the GPU idle while waiting for data; when the CPU performs part of the attention computation, its weak parallelism becomes a new bottleneck, ultimately resulting in low GPU utilization.

1. GPU-CPU collaborative block-level sparse attention: Select key context blocks based on semantic importance; the GPU handles high-priority tasks, while the CPU processes sparse blocks.
2. Pre-layer CPU precomputation: When the GPU processes layer L, the CPU precomputes the sparse attention results for layer L+1, hiding CPU computation latency.
3. Asynchronous periodic recall: Periodically evaluate the generation state, asynchronously recall KV cache data, and dynamically adjust the frequency and scope.

In evaluations across multiple datasets, ScoutAttention controls accuracy loss within 2.4%; achieves a 2.1x speedup compared to existing offloading methods; significantly reduces GPU memory usage, supporting longer contexts or larger batch sizes.

Reduces hardware thresholds: enterprises can deploy long-context models on mid-range GPUs; improves service throughput: a single server can handle more concurrent requests; supports emerging scenarios such as real-time long document analysis and long video understanding.

The current sparse mode is a heuristic design; future research can explore learning-based dynamic sparse strategies; need to expand to multi-GPU distributed inference scenarios; can study collaborative optimization with technologies such as quantization, pruning, and speculative decoding.