KV-Hierarchy-Lab: A Research Framework for Cache Hierarchy Strategies in Long-Context LLM Inference

KV-Hierarchy-Lab is a research platform for evaluating KV cache hierarchy strategies in long-context LLM inference. It uses a trace-driven simulator to help researchers systematically compare the trade-offs between different cache residency, eviction, quantization, and prefetching strategies. The project is explicitly positioned as a research tool rather than a production-grade inference infrastructure, focusing on trace-based simulation to evaluate strategy behaviors while supporting reproducibility and scalability.

Long-context LLM inference faces unique KV cache challenges:

1. Memory Hierarchy Pressure: GPU High Bandwidth Memory (HBM) has limited capacity, requiring some cache to be offloaded to slower tiers like host memory or NVMe, leading to significant differences in access latency;
2. Dynamic Access Patterns: Cache access during inference is non-uniformly distributed (e.g., long-distance access in RAG scenarios or repeated references to conversation history), making static strategies difficult to optimize;
3. Quantization and Precision Trade-offs: Quantization methods like FP8/INT4 reduce memory usage but may introduce precision loss and dequantization overhead;
4. Prefetching Complexity: Incorrect prefetching wastes bandwidth, and predicting access patterns in long contexts is challenging.

The system architecture of KV-Hierarchy-Lab includes the following core components:

• Workload and Trace Generation: Supports synthetic scenarios (retrieval bursts, periodic reuse, mixed locality, adversarial bursts) and importing real traces;
• Simulation Engine: Uses KV pages as the basic unit to simulate page movements between multi-tier memory (Tier0: GPU HBM, Tier1: GPU Overflow Area, Tier2: Host Memory, Tier3: NVMe-like);
• Strategy Interface: Provides pluggable baseline strategies (LRU, windowed_recency, heavy_hitter, cost_aware, predictive, regret_aware);
• Quantization Model: Supports quantization schemes like FP16/FP8/INT4/INT2, considering storage usage and dequantization overhead;
• Benchmarking Tools: Outputs JSON/CSV data and visual charts to support data analysis.

Key research findings from the project using synthetic traces:

1. Advantages of Regret-Aware Strategy: In the rag_burst workload, the number of misses decreased from 212 to 152 (a 26.3% reduction), and latency in adversarial burst scenarios dropped from 3.664ms to 3.365ms;
2. Complexity of Prefetching: Although prefetching reduces misses, speculative traffic may offset gains (e.g., latency in the prefetch_friendly workload is still higher than cost_aware);
3. Dominant Role of Quantization: Switching from FP16 to INT4 increased the hit rate of rag_burst from 0.459 to 0.771, reducing traffic by 93.9%;
4. Strategy Boundaries: Regret-Aware and LRU performed similarly in the chat_continuation scenario.

Evaluation Metrics: Covers multi-dimensional metrics such as overall/hierarchical hit rate, number of misses, average latency, data movement volume, and prefetch efficiency; Application Scenarios and Users: Targets system researchers (exploring new algorithms), inference engine developers (validating strategies), hardware architects (evaluating memory configurations), and quantization researchers (trading off cost and benefit); Typical Workflow: Define/import traces → Configure hierarchy and strategies → Run simulation → Analyze results → Iterative optimization.

Limitations: Based on synthetic traces rather than real runtime data; simulates latency instead of GPU profiling; does not integrate production engines like vLLM; simplified CXL modeling; Future Directions: Import real runtime traces; calibrate with production inference engines; more detailed modeling of host memory and CXL backends.

Industry Insights:

1. Simple LRU is sufficient for most scenarios; complex strategies only show significant advantages in specific patterns;
2. Quantization takes priority over strategy optimization in resource-constrained scenarios;
3. Prefetching needs to be workload-aware to avoid side effects;
4. Multi-dimensional evaluation is required instead of relying on a single metric; Summary: KV-Hierarchy-Lab provides a systematic research tool for KV cache management in long-context LLM inference. Its strategy trade-off analysis has important guiding significance for inference engine development and is a key platform to promote progress in this field.