NeurIPS 2026 Cutting-Edge Research: Quantifying Reasoning Redundancy in the Chain of Thought of Large Language Models

This paper from NeurIPS 2026 proposes an information bottleneck-based framework to quantify Chain of Thought (CoT) efficiency using the Reasoning Information Gain (RIG) metric. It finds that the reasoning process exhibits a three-stage structure: rapid accumulation phase, diminishing returns plateau phase, and convergence phase. This enables 30-53% token compression with an accuracy drop of less than 2%. The study provides a theoretical foundation and practical methods for optimizing LLM reasoning efficiency.

In recent years, large reasoning models like DeepSeek-R1 have improved performance on complex tasks by generating extended Chain of Thought (CoT), but their computational cost is extremely high (the number of reasoning tokens is 5-20 times more than direct answers). Existing studies point out the phenomena of "thought hallucination" and "overthinking". The core questions are: What is the minimum number of reasoning tokens needed to achieve the target answer quality? How to identify and eliminate redundant tokens?

The study proposes the first information-theoretic framework for CoT reasoning efficiency, which includes:

1. Reasoning Information Gain (RIG)：Measures the contribution of each token to reducing answer uncertainty, with the formula $\text{RIG}(t) = H(A \mid x, r_{<t}) - H(A \mid x, r_{1:t})$；
2. Cumulative Reasoning Information (CRI)：$\text{CRI}(t) = \sum_{i=1}^t \text{RIG}(i)$, and reasoning efficiency $\eta(t)=CRI(t)/CRI(T)$；
3. Reasoning-Specific Lower Bound：Using the semantic decomposition structure of CoT, a minimum effective length lower bound that is 1.8-3.2 times tighter than the general bound is obtained.

1. Three-Stage Structure: Across all models/tasks, there exists a rapid information accumulation phase (first 15-25% of tokens, contributing 60-70% of information), a diminishing returns plateau phase (middle 40-70% of tokens, contributing <15% of information, main source of waste), and an answer synthesis convergence phase (last 10-25% of tokens);
2. Redundancy Quantification: Specialized reasoning models (e.g., DeepSeek-R1) have 1.8-2.3 times longer chains than general models, but their minimum effective lengths are comparable, leading to higher redundancy rates (55-66% vs. 50-59% for general models);
3. Estimator Guarantee: The RIG estimator $\widehat{RIG}(t)$ based on next-token distribution shift has a small gap from the true value (coupling divergence <0.3 nats for 87% of tokens).

An early stopping criterion is designed based on the three-stage structure: detect the transition from the accumulation phase to the plateau phase via window-averaged RIG, then stop and generate the answer. Experimental results: 30-53% token savings are achieved on datasets like GSM8K and MATH, with an accuracy drop of <2%, outperforming 5 baseline methods such as fixed truncation and entropy thresholding.

• Model Design: Current training overemphasizes detailed explanations; future work can introduce RIG regularization to reduce redundancy; dynamically allocate reasoning budgets (simple questions only need tokens from the accumulation phase); plateau phase redundancy supports latent reasoning;
• Information Bottleneck Extension: Extend the traditional information bottleneck from network layers to the temporal token generation domain;
• Test-Time Computation: The diminishing returns in the plateau phase suggest that information efficiency should be considered instead of just increasing length.

Limitations: Based on the greedy decoding assumption; validation tasks are limited to math, scientific reasoning, etc.; experiments use 7B models, and the behavior of larger-scale models remains to be verified; Future Directions: Adaptive reasoning architecture (dynamically adjust depth); extension to multimodal reasoning; human-machine collaborative reasoning (human intervention at key nodes); further tightening of theoretical lower bounds.