New Approach to KV Cache Compression: A Minimal-Intervention Diversity Penalty Strategy

This article addresses the bottleneck of KV cache memory usage in large language model inference. After systematically evaluating seven existing compression mechanisms (none of which passed strict validation), we propose a minimal-intervention method called Alpha—by introducing a diversity penalty strategy based on the facility location problem into KV selection, significant results are achieved with only a single function modified. This method has been validated through pre-registered experiments, proving effective under specific model and budget conditions, and the simple improvement outperforms complex structural redesigns.

The efficiency bottleneck of large language model inference stems from the linear growth of KV cache memory usage with sequence length, creating an urgent need for compression in resource-constrained scenarios. However, the design space of KV cache compression is complex (covering multiple dimensions such as representation methods and routing strategies), making it difficult for researchers to identify effective improvements. This study pre-registered and evaluated seven mechanisms across five families, none of which passed statistical tests, revealing that there may be a large number of "false positive" results in the field.

The Alpha method makes minimal modifications to the existing TriAttention retention scorer: replacing argmax-top-k with a greedy selection strategy based on the facility location problem, and introducing a redundancy penalty term controlled by λ. The implementation steps are: calculate KV importance scores → iteratively select KVs that maximize marginal gain (considering similarity redundancy with the selected set). The best performance is achieved when λ=0.5, balancing accuracy and diversity.

The experiment uses the mathematical reasoning task (MATH-500 dataset) as the benchmark (requiring long-range dependencies and high KV quality), employing the DeepSeek-R1-Distill inference models of Qwen-7B and Llama-8B, and focusing on small budget scenarios of 64/128. In the pre-registration protocol, λ is tuned on the development set and validated on the test set, requiring passing Bonferroni-corrected multiple tests. Results: When λ=0.5, Qwen (b=128) and Llama (b=64) passed the tests with no significant negative results.

The most significant finding of the study is asymmetry: the Alpha method, which only modifies the scoring function, outperforms seven more complex structural redesigns. This challenges the assumption that "larger architectural changes are necessarily better". The core insight is the importance of diversity penalty—retaining diverse information under limited budget is more critical than selecting a single optimal option. Strict pre-registration and statistical tests make this finding evident.

Limitations: Only some test conditions passed strict tests; effectiveness may depend on model/task characteristics; limited to mathematical reasoning tasks, and applicability to other tasks (e.g., code generation) remains to be verified. Future directions: Adaptive adjustment of λ parameters; exploration of combinations with techniques like quantization/pruning; validation of effects on larger models.

Implications include: 1. Strict evaluation (e.g., pre-registration, statistical tests) is key to distinguishing real progress from false signals; 2. Value of minimal intervention: simple and interpretable methods are often more practical than complex black-box solutions; 3. Importance of information diversity under resource constraints, which can be extended to other compression/selection problems.