DUAL-BLADE: A Dual-Path KV Cache Offloading Framework for Edge Devices

This article introduces DUAL-BLADE, a dual-path KV cache residency framework for edge AI systems. By dynamically allocating KV tensors to either the page cache path or the NVMe direct access path, the framework bypasses file system overhead to achieve low-latency direct storage access, reducing latency by 33.1% in the prefill phase and 42.4% in the decoding phase. It aims to address the problem of limited memory resources in LLM inference on edge devices.

When deploying large language models (LLMs) to edge devices, edge devices have limited memory resources, and KV cache (Key-Value Cache) is a major memory consumer, often exceeding available memory in long-context scenarios. Traditional file-based offloading designs rely on the kernel page cache, which leads to cache thrashing, unpredictable latency, and high software overhead under memory pressure—problems that are more pronounced in edge environments.

The core idea of DUAL-BLADE is to dynamically select the optimal access path based on runtime memory availability: use the page cache path when memory is sufficient (leveraging the OS's mature caching mechanism); switch to the NVMe direct access path when memory is tight (bypassing the file system, directly mapping KV tensors to contiguous LBA regions to achieve low-overhead direct storage access), thus flexibly adapting to resource conditions.

1. Direct access bypassing the file system: Eliminates overheads such as path resolution, permission checks, metadata management, and page cache replacement policies; 2. Contiguous LBA mapping: Enables sequential read optimization, reduces seek time, and simplifies address calculation; 3. Adaptive pipeline parallelism: Overlaps storage I/O with GPU DMA operations, dynamically adjusts pipeline depth to hide I/O latency and improve throughput.

Evaluations show: Latency is reduced by up to 33.1% in the prefill phase and up to 42.4% in the decoding phase; SSD utilization is increased by 2.2x; these benefits remain stable across various memory budget configurations, making it suitable for edge deployment.

1. Reduces hardware costs: Makes it possible to run LLMs on lower-configured hardware; 2. Improves user experience: Faster first token generation and smooth streaming output, benefiting latency-sensitive applications; 3. Extends device battery life: Efficient I/O reduces storage active time and lowers power consumption.

1. Storage device selection: It is recommended to use NVMe SSDs with high concurrent I/O and low latency; 2. Memory-storage trade-off: Retain more memory for latency-sensitive scenarios, and actively offload for cost-sensitive ones; 3. Integration with existing systems: Can be integrated with inference frameworks like vLLM and TensorRT-LLM, and its modular design makes it easy to introduce optimizations.

Current limitations: Mainly supports a single NVMe device; Future directions: Optimization of KV cache distribution in multi-storage device environments, combination of KV compression and offloading, and predictive prefetching based on access patterns.