EGRSD: Enhancing Reasoning Efficiency of Large Language Models via Entropy-Aware Self-Distillation

This article proposes the EGRSD (Entropy-Guided Reinforced Self-Distillation) method, which dynamically adjusts the supervision weights of each position in the reasoning chain through an entropy confidence gating mechanism from the teacher model to address the uniform weighting problem in existing self-distillation methods. This method optimizes reasoning length while maintaining accuracy and has been validated effective on the Qwen3 model; it also introduces the CL-EGRSD causal look-ahead variant to further refine the supervision signals. This article will discuss aspects such as background, methodology, experiments, and significance.

In recent years, the reasoning capabilities of large language models have made significant progress. Self-distillation technology allows models to learn from their own reasoning trajectories, with token-level supervision provided by the teacher model. However, existing methods assign the same weight to all tokens, ignoring changes in the entropy of the teacher's prediction distribution—some positions are certain for the model, while others are highly uncertain. Uniform weighting treats noise signals and reliable signals equally, which has become a key challenge for efficiency improvement.

The EGRSD method addresses the uniform weighting problem through an entropy confidence gating mechanism. The token-level update is the product of three signals:

1. Reward-oriented signal: Guides the direction based on task rewards (e.g., answer correctness) to align with training objectives;
2. Magnitude of teacher-student likelihood ratio: Measures the prediction difference between teacher and student models; a larger difference requires a greater update magnitude for the student;
3. Teacher entropy confidence gating (core): Dynamically adjusts weights based on the teacher's prediction entropy—high weights for low-entropy (certain) positions, low weights for high-entropy (uncertain) positions, with a non-zero lower bound to avoid ignoring steps.

The paper proposes the CL-EGRSD variant, which distinguishes two types of high-entropy positions:

• Persistent high entropy: The entire reasoning segment is difficult, with consecutive uncertain positions;
• Transient high entropy: Temporarily uncertain, with clear subsequent context. Through the causal look-ahead mechanism, the subsequent context of high-entropy positions is observed: if the subsequent entropy becomes low, the current weight is increased; if the high entropy persists, the weight remains low, making the supervision signal more precise.

Experiments were conducted on Qwen3-4B and Qwen3-8B models, and the results show:

• Accuracy-length frontier improvement: Performs better than existing methods on the accuracy-length trade-off curve; can maintain/improve accuracy while shortening the reasoning chain, or achieve higher accuracy at the same length;
• Efficiency advantage: Avoids resource waste on high-uncertainty positions, making training more efficient;
• Generalization ability: Consistent performance across models of different scales, indicating good generalization of the entropy-aware mechanism.

Significance of EGRSD:

• Theoretical level: Reveals that model uncertainty estimation can serve as an effective learning signal, providing new ideas for self-supervision and curriculum learning;
• Practical level: Lightweight improvement, no additional models or architecture modifications needed—only adjusts the loss function weights, easy to integrate into existing workflows;
• Efficiency level: Optimizes the accuracy-length trade-off, reducing deployment costs (shorter reasoning chain → lower latency and overhead).

Limitations:

1. Experiments were mainly conducted on Qwen3 models; applicability to other architectures (e.g., GPT, LLaMA) needs to be verified;
2. Entropy gating hyperparameters (e.g., threshold, lower bound) need to be tuned for different tasks. Future directions:

• Extend to multimodal reasoning;
• Explore more complex causal look-ahead window strategies;
• Combine with other reinforcement learning variants (e.g., PPO, GRPO).

EGRSD assigns more intelligent supervision signals to self-distillation through an entropy-aware mechanism, focusing on the reasoning steps where the model truly needs help to achieve efficient capability improvement. It reminds us that training reasoning models not only requires attention to 'what to learn' but also 'where to learn', concentrating resources on key steps.