MoE-nD: Achieving 14x KV Cache Compression with Hierarchical Mixture-of-Experts Strategy While Preserving Long Text Inference Performance

MoE-nD breaks through the bottleneck of traditional uniform compression methods by customizing differentiated KV cache compression strategies for different Transformer layers. It maintains the original model performance even at a 14x compression ratio, paving the way for the practical application of long-text large language model inference.

As the context window of large language models expands to hundreds of thousands or even millions of tokens, the memory footprint of KV cache has become a major bottleneck for inference efficiency. Existing compression methods (such as token eviction, quantization, low-rank projection, etc.) apply the same strategy to all Transformer layers, ignoring the differentiated responses of layers to compression operations, which results in suboptimal model quality under the same memory budget.

The core insight of MoE-nD is that different Transformer layers have significant differences in sensitivity to compression operations. The technical implementation is divided into two phases: the offline calibration phase uses a greedy solver to select the optimal (eviction rate, K-bits, V-bits) configuration for each layer; the runtime phase applies hierarchical heterogeneous eviction and quantization strategies through a unified attention patch—for example, the first layer uses a 90% token retention rate + 8-bit quantization, while deeper layers use a 70% retention rate + 4-bit quantization.

In 4 task subsets of LongBench-v1 (16K input length), MoE-nD fully matches the uncompressed baseline performance at 14x compression (1.9GB → 136MB); other baseline methods score less than 8/100 under equivalent or smaller memory footprints. On the AIME inference benchmark, MoE-nD outperforms the strongest uniform quantization baseline by 6-27 percentage points, with no significant improvement in short-text tasks—verifying its value in long-text scenarios.

MoE-nD reveals the principle of neural network heterogeneity: different Transformer layers have unique 'characteristics', and ignoring heterogeneity with a uniform strategy wastes optimization space. This idea can be extended to techniques such as pruning, distillation, and sparsification, providing a feasible path and empirical basis for related research.

Limitations: No significant improvement in short-text tasks (e.g., MATH-500, TREC). Future directions: Dynamically adjust strategies to adapt to different input lengths, extend to the attention head level, and combine with efficient inference techniques such as speculative decoding.