Cascade: Breaking GPU Memory Limits, Extending Large Model Context Windows with Disk KV Caching

The Cascade project proposes an innovative disk KV caching technology. By leveraging the storage hierarchy of GPU memory, system memory, and disk, it solves the GPU memory bottleneck caused by the linear growth of KV cache with context length in the Transformer architecture. This enables significant expansion of the context window for large language models, supporting ultra-long context scenarios such as long document processing and codebase analysis.

Long Context Demand

Extending the context window of large language models can support scenarios like whole book processing, multi-turn deep conversations, and large codebase analysis, but it faces GPU memory constraints.

Memory Issue of KV Cache

In the Transformer self-attention mechanism, the KV cache grows linearly with sequence length:

• The size of KV pairs per token = 2 × hidden dimension × precision bytes
• For a 70B model in FP16, the KV cache for 100K tokens is approximately 3.2GB (single layer, single head), and actual models require tens to hundreds of GB of memory.

Three-Tier Storage Architecture

1. GPU Memory (Hot Cache)：Stores recently used KV pairs with nanosecond-level latency
2. System Memory (Warm Cache)：Stores less frequently accessed KV pairs with microsecond-level latency
3. Disk Storage (Cold Cache)：Stores historical KV pairs with TB-level capacity

Intelligent Strategy

• LRU Replacement: Evicts the least recently used KV pairs when GPU memory is full
• Prefetching: Loads potentially needed KV pairs in advance
• Block Storage: Fine-grained migration reduces overhead
• Compression Encoding: Reduces disk I/O and storage usage

Technical Implementation

• Serialization: Zero-copy, memory mapping, asynchronous I/O
• Random Access: Index structure, block alignment, Bloom filter
• Consistency: Write-back strategy, version control, crash recovery

Performance Characteristics

• Optimal Scenario (Good Locality): GPU hit ~0.1ms/token, memory hit ~0.5ms/token
• Challenging Scenario (Long-Distance Dependencies): Disk hit ~5-10ms/token

Application Scenarios

1. Long novel generation
2. Codebase-level analysis
3. Multi-document Q&A
4. Unlimited conversation history
5. Long video understanding

Comparison with Existing Technologies

• Sparse Attention: Requires retraining, may lose long dependencies
• Sliding Window: Loses context outside the window
• Model Compression: Affects computation quality Cascade maintains full attention and only changes storage locations.

Cascade is a practical innovation to solve the context limitations of LLMs. It does not change the attention mechanism but uses a mature storage hierarchy to break through GPU memory limits, supporting next-generation AI applications (such as whole book reading and codebase understanding), which is a solid step toward general artificial intelligence.

Current Limitations

1. I/O Bottleneck: High disk access latency
2. Increased Power Consumption: Frequent disk I/O
3. Increased System Complexity
4. Dependence on high-speed SSD and PCIe bandwidth

Future Directions

• Intelligent Prefetching: Precise preloading based on attention patterns
• Hierarchical Compression: High precision for hot data, aggressive compression for cold data
• Distributed Expansion: Multi-node storage of KV cache
• Dedicated Hardware: Optimize memory expansion using CXL technology