KVDrive: A Comprehensive Multi-Level KV Cache Management System for Long-Context LLM Inference

Long-context LLM inference faces a bottleneck where KV cache memory requirements grow linearly with sequence length. KVDrive provides a system-level solution for long-context inference by achieving a 1.74x throughput improvement while maintaining accuracy through multi-level cache management across GPU memory, host memory, and SSD, combined with attention-aware cache placement and pipeline scheduling optimizations.

As LLM capabilities expand, long context has become a key indicator of model practicality, but KV cache memory requirements grow linearly with sequence length, quickly becoming a bottleneck for inference systems. Existing offloading systems alleviate this via sparsification strategies, but sparsity cannot be pushed indefinitely without compromising accuracy. Moreover, as context length and batch size increase, the surge in KV transfer data volume leads to decoding delays. This reveals that pure algorithmic optimization cannot solve systemic challenges—KV cache management needs to be rethought from a system architecture perspective.

KVDrive is a comprehensive multi-level KV cache management system covering three tiers: GPU memory, host DRAM, and SSD. Unlike previous work focusing on algorithmic sparsification, it maintains high-throughput inference under tight GPU budgets through system-level coordinated orchestration of cache placement, pipeline scheduling, and cross-tier coordination. Its core design philosophy treats KV cache as a resource across multiple storage tiers to balance performance and capacity.

KVDrive achieves breakthroughs in three dimensions:

1. Attention-Aware Cache Management: Identify frequently accessed hot KV data to keep in GPU memory, predict future access patterns for prefetching, maximize reuse, and reduce redundant data movement;
2. Pipeline Reconstruction and Compute-Communication Overlap: Redesign the decoding pipeline to overlap I/O-intensive (cross-tier data retrieval) and compute-intensive (attention calculation) phases, eliminating resource idle waiting;
3. Cross-Tier Data Movement Coordination: Treat multiple storage tiers as a unified pool, coordinate data flow via intelligent replacement, prefetching, and compression strategies to unlock scalable long-context inference capabilities.

KVDrive's implementation includes three core components:

1. Hierarchical Cache Abstraction Layer: Provides a unified interface to access KV caches across different tiers, shielding underlying complexity;
2. Asynchronous Data Transfer Engine: Leverages asynchronous transfer capabilities like GPUDirect Storage to enable efficient data movement and decouple it from computation;
3. Adaptive Scheduler: Dynamically adjusts strategies based on workload characteristics to optimize cache placement and prefetching decisions online.

The research team evaluated KVDrive on real-world long-context benchmarks using multiple popular LLM models. Results show that it achieves up to a 1.74x throughput improvement compared to state-of-the-art work while maintaining accuracy. This demonstrates the great potential of system-level optimization, and the architecture has good scalability to support longer contexts as storage technology advances.

KVDrive brings three technical insights:

1. System-Level Thinking: In resource-constrained scenarios, we need to coordinate interactions between components to achieve global benefits;
2. Workload Characteristics: Optimizations based on the predictability and locality of attention mechanisms are more effective;
3. Heterogeneous Collaboration: Efficient collaboration between heterogeneous components like CPU and GPU is a future direction. Looking ahead, the importance of long-context inference will become more prominent, and KVDrive's system-level optimization approach is expected to become a standard paradigm for next-generation inference infrastructure.