ReasonAlloc: A New Paradigm for KV Cache Budget Allocation in Reasoning Models, Breaking the Memory Bottleneck of Long-Chain Reasoning

ReasonAlloc is a training-agnostic hierarchical budget allocation framework proposed to address the KV cache explosion problem in long-chain thinking reasoning of reasoning models. Through offline layer pre-allocation (capturing the "reasoning wave" pattern) and online head reallocation (dynamic resource optimization) strategies, it significantly reduces KV cache pressure, especially in small-budget scenarios. It is compatible with existing compression methods and has negligible inference overhead.

Long-chain thinking reasoning leads to linear expansion of KV cache, which becomes a bottleneck. Limitations of existing solutions: Compression during decoding assumes uniform layer/head requirements; non-uniform allocation targets the static prompt phase and cannot adapt to the dynamic needs of autoregressive reasoning. Core question: How to dynamically and intelligently allocate limited KV cache budget during the decoding phase?

Offline Layer Pre-Allocation

Identify the "reasoning wave" pattern (regular fluctuations in layer requirements), and allocate differentiated budgets to each layer through architecture analysis and a small amount of calibration data, without training.

Online Head Reallocation

Real-time monitoring of attention head information density, using lightweight heuristic metrics to evaluate utility, and dynamically transferring resources to high-utility heads.

• Training-Agnostic: No fine-tuning required; directly applicable to off-the-shelf models.
• Compatible with Existing Strategies: Can be combined with compression methods like R-KV and SnapKV.
• Low Inference Overhead: Additional latency is negligible.
• Hierarchical Decoupling: Offline processing handles inter-layer differences; online processing addresses intra-head dynamic changes.

Test models include DeepSeek-R1-Distill-Llama-8B, etc. Compared with the uniform budget baseline, the findings are:

1. The most significant improvement in small budgets of 128-512 tokens;
2. Steadily outperforms all baselines;
3. Consistent improvement patterns across 8B-14B models;
4. No sacrifice in reasoning accuracy.

• Reduce Deployment Costs: The same hardware supports longer reasoning chains or cheaper configurations;
• Support Edge Deployment: Memory-constrained devices can run high-quality reasoning models;
• Promote Technology Popularization: Lower the threshold for reasoning to accelerate democratization;
• Inspire New Directions: Extend the hierarchical allocation idea to mobile/real-time systems.

ReasonAlloc solves the KV cache bottleneck through hierarchical budget allocation, requiring no training and showing prominent effects in small-budget scenarios. Its core is understanding the demand differences of reasoning components, providing key infrastructure for efficient deployment of reasoning models, and will promote technology implementation.