EntropyInfer: An Entropy-Guided Adaptive Inference Framework for Large Models on Long Texts

Core Information

• Project Name: EntropyInfer (Entropy-Guided Adaptive Inference Framework for Large Models on Long Texts)
• Core Method: Dynamically identify rigid and dynamic attention heads via attention entropy, enabling head-level and segment-level adaptive computation allocation
• Main Results: Achieve a 2.39x end-to-end speedup on long texts with over 100,000 tokens, with minimal quality loss
• Source & Open Source: arXiv paper (published on June 8, 2026, link: http://arxiv.org/abs/2606.09508v1), code open-sourced at https://github.com/SHA-4096/EntropyInfer

Efficiency Bottlenecks

When large language models process long texts, attention computation and KV cache storage are the main bottlenecks.

Flaws of Existing Methods

Sparse attention and KV cache compression methods have the problem of a "one-size-fits-all" strategy:

• Apply the same sparse pattern to all attention heads
• Use a uniform computation budget for different contexts
• Ignore differences in attention behavior between heads and across contexts Leading to inefficient resource allocation.

Role of Entropy

Attention entropy measures distribution uncertainty: low entropy (focused on a few positions), high entropy (scattered browsing).

Two Types of Attention Heads

• Rigid Heads: Entropy value close to zero, fixed behavior (e.g., position encoding, syntax marker heads)
• Dynamic Heads: Entropy value fluctuates, adjusts focus with context (e.g., semantic content, entity association heads)

Key Finding

The distribution of head types is context-dependent and cannot be pre-determined offline.

Prefill Phase

• Head-Level Allocation: More resources for high-entropy heads, aggressive compression for low-entropy heads
• Segment-Level Allocation: Split long inputs into segments, adjust strategies independently for each segment

Decoding Phase

• Consider KV cache compression of generated output tokens
• Compress KV cache in latent space to reduce memory usage

Model Benchmarks

Tested on Llama, Qwen, and openPangu series models.

Main Results

• End-to-End Speedup: Up to 2.39x in scenarios with over 100,000 tokens
• Baseline Comparison: Outperforms SnapKV, AdaKV, and CritiPrefill
• Quality Preservation: QA accuracy loss <2%, summary ROUGE>98%, code generation Pass@1 shows almost no drop.

Application Scenarios

• Long Document Processing: Legal contracts, academic papers, book summaries
• Dialogue Systems: Customer service bots, personal assistants, education tutoring
• Code Generation: Code completion, review, document generation

Open Source Contributions

Code is open-sourced at https://github.com/SHA-4096/EntropyInfer, including core implementation, multi-model adaptation, evaluation scripts, and usage documentation.

Limitations

• Entropy computation introduces additional overhead
• Some optimizations depend on specific hardware
• Effectiveness for extreme lengths (million tokens) needs verification

Future Directions

• Hardware co-design
• Theoretical deepening (link between entropy and model capability)
• Multimodal extension
• Fully adaptive computation system

Conclusion

EntropyInfer breaks through the efficiency bottleneck of long text inference, realizes intelligent resource allocation by understanding attention behavior, and adaptivity is the future direction.