VeriCache: A Validation Framework for Transforming Lossy KV Cache into Lossless LLM Inference

VeriCache addresses the cumulative error issue of traditional KV cache compression in long sequence generation. Its core approach uses compressed KV cache to draft tokens and full KV cache for verification, ensuring output consistency with full-precision inference while achieving up to 4x throughput improvement, solving the dilemma between memory cost and quality risk.

As LLM context length grows, KV cache memory becomes a key inference bottleneck. Existing compression methods (token dropping, quantization, pruning) save memory but are lossy—cumulative errors in long sequences lead to catastrophic failures in high-accuracy scenarios like code generation. This creates a dilemma: choose full KV cache's high cost or lossy compression's quality risk.

VeriCache's core is "draft first, verify later": use compressed KV cache to generate candidate tokens (draft phase) then verify with full KV cache (validation phase), only accepting verified tokens to ensure consistency with full KV cache output. Similar to speculative decoding but faces a unique challenge: full KV cache needs dynamic swapping from external storage, so minimizing swap overhead is critical.

Two key insights power VeriCache: 1. Parallel heterogeneous operations: Compressed KV decoding (HBM bandwidth-limited) and full KV swap (PCIe/network bandwidth-limited) can run in parallel (GPU generates candidates while preloading full KV cache). 2. Long draft window amortization: Compressed cache outputs are similar to full cache, so long draft windows spread swap costs over multiple verifications, reducing average overhead.

VeriCache is a flexible framework: 1. Unified compressor interface: Supports various KV compression methods via a standard interface. 2. Multi-scene adaptation: Works for long context decoding (model's own KV cache) and remote prefix cache (external KV data verification). 3. Synergy with speculative decoding: Combines with small-model speculative decoding as a multi-layer acceleration strategy.

Real workload evaluations show VeriCache achieves up to 4x throughput improvement while maintaining output consistency with full KV cache inference. This proves KV compression doesn't need to sacrifice quality, making it valuable for production LLM services.

Insights: 1. "Approximate computation + precise verification" is applicable in resource-constrained scenarios. 2. System optimization requires hardware understanding (HBM/PCIe heterogeneity, GPU parallelism).3. Universal interfaces reduce adoption costs. Future: As context length grows and multi-modal apps spread, VeriCache's validation paradigm may become a standard component of next-gen inference systems.