MBE Protocol: Establishing a Standardized Evaluation System for KV Cache Compression in Large Models

The Matched-Budget Evaluation (MBE) protocol is a standardized fixed-budget reporting protocol and open-source evaluation framework for KV cache compression methods in large language models. It aims to resolve the fragmented issue of incomparable evaluation results in the current KV cache compression field. Its core idea is to compare methods under the same reserved KV memory budget. Through fixed budget tiers and a multi-dimensional evaluation matrix, different research results can be directly compared.

In LLM inference, KV cache is the main source of memory consumption, and its linear growth with sequence length becomes a bottleneck. Although there are various compression methods such as quantization and pruning, different studies use different models, tasks, and metrics, and even lack systematic measurement, leading to results that cannot be directly compared, making it difficult for researchers and engineers to select appropriate methods.

The core of MBE is to compare methods under the same reserved KV memory budget. It is not a new benchmark but a lightweight reporting layer that is compatible with existing task suites (such as LongBench, GSM8K, etc.). It defines fixed budget tiers: B50 (50%), B25 (25%), B12 (12.5%), B06 (6.25%, optional), which facilitates observing performance curves under different compression intensities.

MBE requires reporting multi-dimensional metrics at each budget point:

• Model dimension: Covers 7-8B GQA, 7-14B, and ≥70B models
• Task dimension: Retrieval, aggregation/tracking, instruction following, reasoning, agent/multi-turn tasks
• System dimension: Peak memory, throughput, first token time, maximum batch size, hardware level
• Method dimension: Deployment prerequisites (training-free/calibration/pretraining), composability.

MBE provides an adapter-based open-source framework. Researchers only need to implement the KVCompressor interface, and the framework automatically handles budget scanning, task execution, and metric collection. Built-in reference adapters include KIVI (2-bit quantization), H2O (dynamic eviction), SnapKV, StreamingLLM, PyramidKV, etc., which lowers the evaluation threshold.

MBE adopts an open contribution model. Researchers can submit evaluation cards (via PR), and CI automatically updates the leaderboard. Quick start steps:

1. Configure methods and running parameters using YAML
2. Run run_mbe.py to generate evaluation cards
3. Render the cards and submit a PR.

MBE not only solves the fragmented problem of KV cache compression evaluation but also represents a new paradigm for scientific research collaboration. Industry can select methods objectively, and academia can lower the evaluation threshold. As LLM context windows expand, the importance of KV compression increases, and MBE is expected to become the infrastructure in this field, promoting more comparable and reproducible research.