EKVA: Expert-Aware KV Cache Budget Allocation Optimization Scheme for Sparse MoE Large Models

Introducing the EKVA project, which achieves expert-aware KV cache budget allocation in sparse MoE large language model inference through Roofline model-guided Triton kernel optimization, significantly improving inference efficiency.

• Original Author/Maintainer: GauravPatil2515
• Source Platform: GitHub
• Original Title: EKVA
• Original Link: https://github.com/GauravPatil2515/EKVA
• Publication Date: 2026-06-11

Mixture of Experts (MoE) has become an important paradigm for scaling current large language models (LLMs). Unlike dense models, MoE models only activate a subset of experts during inference, thus reducing computational costs while maintaining model capacity. Typical MoE architectures such as Mixtral and Qwen-MoE have demonstrated strong performance.

However, MoE models face a unique challenge during inference: memory management of Key-Value (KV) cache.

During the autoregressive generation process of Transformers, the model needs to cache the Key and Value vectors of previous tokens to avoid redundant computations. This caching significantly accelerates the generation process but also introduces huge memory overhead:

• For long sequences, KV cache may occupy most of the GPU memory
• In batching scenarios, cache requirements grow linearly with batch size
• The sparsity of MoE models makes cache management more complex

In MoE models, each token is usually routed to only a few experts (e.g., 2). This means:

• Different tokens activate different combinations of experts
• Traditional uniform KV cache allocation strategies lead to memory waste
• Cache needs to be managed separately for each expert, increasing complexity

EKVA (Expert-Aware KV Budget Allocation) proposes an expert-aware KV cache budget allocation strategy, which uses Roofline performance model-guided optimization to achieve the optimal balance between memory and computational efficiency.

The core insight of the project is: In MoE inference, different experts have different levels of importance.

• Some experts are activated more frequently
• Some experts contribute more to the final output
• Therefore, more KV cache budget should be allocated to important experts

EKVA uses the Roofline performance model to analyze inference bottlenecks:

1. Computational Bottleneck vs. Memory Bottleneck: The Roofline model helps identify whether computation or memory bandwidth is the bottleneck under the current configuration
2. Optimal Configuration Search: Based on Roofline analysis, search for the optimal KV cache allocation strategy
3. Hardware-Aware Optimization: Consider the memory hierarchy of specific GPU architectures (e.g., A100, H100)