Entropy-Gradient Reversal: Deep Dive into the Internal Mechanisms of Large Reasoning Models

A paper published in May 2026 identifies a robust negative correlation between token entropy and logit gradients in Large Reasoning Models (LRMs) — termed "Entropy-Gradient Reversal" — and defines it as a geometric fingerprint of reasoning ability. Based on this discovery, the research team proposes the CorR-PO method, which embeds this reversal feature into the reward regularization of Reinforcement Learning (RL). It achieves better performance than existing methods on multiple reasoning benchmarks and improves training stability.

Current research on large reasoning models faces two key challenges:

1. Gap between behavioral analysis and internal mechanisms: Existing analyses mostly stay at the token-level behavioral level, lacking a systematic understanding of the model's internal reasoning mechanisms, with limitations such as superficiality, lack of causality, and non-generalizability;
2. Instability of RL optimization: RL methods relying on external validators are costly and have sparse rewards, leading to unstable training processes, which are prone to local optima or performance collapse.

Entropy-Gradient Reversal refers to the negative correlation between token entropy (the model's uncertainty about the next token prediction) and logit gradients (the sensitivity of outputs to parameters): when entropy is low (model is certain), gradients are high; when entropy is high (model is uncertain), gradients are low. This feature is called a geometric fingerprint because it has:

• Robustness: Consistently exists across different model sizes, architectures, and datasets;
• Discriminability: The strength of the reversal is positively correlated with reasoning performance;
• Measurability: Can be obtained from internal states without external validation.

The core of CorR-PO (Correlation Regularized Population Policy Optimization) is to add the Entropy-Gradient Reversal feature as a reward regularization term to RL optimization:

• Base reward: Task reward based on external validators;
• Regularization term: Encourages the model to exhibit a stronger Entropy-Gradient Reversal feature;
• Optimization objective: L = L_task + λ*L_reg (L_task is the task loss, L_reg is the reversal correlation regularization loss, λ is the regularization coefficient).

In benchmark tests such as mathematics (GSM8K, MATH), logic, and multi-step reasoning:

1. CorR-PO consistently outperforms existing methods like GRPO and PPO;
2. The strength of the reversal is positively correlated with reasoning performance;
3. The training process is more stable, reducing the risk of performance collapse. Ablation experiments show that an appropriate regularization coefficient improves performance, and CorR-PO has good generalization across different model sizes.

Theoretical Contributions:

• First formal definition of the Entropy-Gradient Reversal phenomenon;
• Provides an intrinsic evaluation metric for reasoning ability (without external validation). Methodological Contributions:
• Proposes the CorR-PO method, combining internal mechanisms with external reward optimization;
• Reduces reliance on external validators and lowers training costs. Insights:
• The importance of in-depth research on internal mechanisms;
• The geometric perspective can reveal patterns not captured by traditional analysis;
• Combining internal and external signals can improve optimization effects.

The current research has the following limitations and future directions:

1. Generality: Need to verify the applicability of Entropy-Gradient Reversal in tasks such as code generation and multimodal reasoning;
2. Theoretical explanation: Need to further study the deep mechanism of the reversal phenomenon;
3. Multimodal extension: Extending this concept to multimodal reasoning models is a potential direction.