Comparison of KV Cache Management Strategies: An Empirical Study of vLLM, InfiniGen, and H2O

This study conducts a systematic comparison of three advanced KV cache management frameworks—vLLM, InfiniGen, and H2O—revealing their performance characteristics under different request rates, model sizes, and sparsity conditions, and providing practical guidance for strategy selection in memory-constrained scenarios.

In large language model inference, KV cache avoids redundant computations and keeps generation complexity linear. However, as model size, context length, and concurrent requests increase, memory usage becomes a bottleneck. Existing strategies such as tensor offloading, token eviction, and speculative scheduling each have their own characteristics, but there is a lack of clear guidance on their advantages and disadvantages under heterogeneous loads and diverse configurations.

vLLM uses paged memory management to reduce fragmentation; InfiniGen handles long contexts through intelligent tensor offloading; H2O retains important tokens based on attention heatmaps. Experiments evaluate latency, throughput, and memory usage, covering dimensions such as request rate, model size, and sparsity.

vLLM performs excellently in medium-sized models and high-concurrency scenarios; InfiniGen is suitable for long-context applications; H2O makes a pragmatic trade-off between quality and resources in extremely memory-constrained environments.

Choose vLLM when resources are sufficient; use InfiniGen for long contexts; use H2O when resources are limited. Dynamic switching or combination of strategies is possible (e.g., full caching for short requests, compression/offloading for long requests).

There is no universal optimal strategy; selection must be based on load and constraints. Current strategies are mostly heuristic, lack task adaptation, and cache management is independent of the inference process. Future research needs to explore more task-adaptive dynamic strategies.