ERPO: Token-level Entropy Regulation Policy Optimization Method for Large-scale Reasoning Models

This article introduces ERPO (Entropy Regulation Policy Optimization), a new method to improve the training of large-scale reasoning models. By identifying Critical Decision Points (CDPs) and introducing three collaborative mechanisms, ERPO addresses the problem of premature entropy collapse caused by uniform advantage allocation in GRPO, achieving higher accuracy and more concise reasoning paths in mathematical reasoning benchmark tests.

In recent years, Reinforcement Learning with Verifiable Rewards (RLVR) has driven progress in the reasoning capabilities of large language models, but the mainstream method GRPO has flaws: assigning uniform advantage values to all tokens, ignoring the heterogeneity of information in the reasoning chain, leading to premature entropy collapse (policy converges to a fixed pattern) and long, low-quality reasoning paths.

The research team identified Critical Decision Points (CDPs) — transient high-entropy states in the reasoning process where the policy trajectory is sensitive to perturbations (e.g., reasoning forks). The uniform advantage signal of GRPO suppresses CDP exploration, making the model tend to take conservative paths rather than optimal strategies.

ERPO shifts the optimization focus to token dynamics and includes three components: 1. Entropy-aware gating mechanism: adaptively identifies CDPs and amplifies exploration intensity; 2. Bucket-based implicit normalization: groups samples by difficulty to alleviate gradient imbalance; 3. Result-anchored advantage synthesis: reweights token signals based on the correctness of the final answer to reflect the contribution of each step to the result.

Experiments on the MATH dataset and AIME competition problems show that: ERPO significantly outperforms the GRPO baseline with improved accuracy; reasoning paths are more concise and robust; it establishes a new Pareto frontier for efficiency and accuracy, proving that high-quality reasoning does not have to sacrifice efficiency.

ERPO brings the following insights: 1. Token-level refined optimization is key to improving reasoning quality; 2. The balance between exploration and exploitation needs dynamic adjustment; 3. Structured credit assignment is crucial for complex reasoning to avoid signal dilution.

ERPO represents an important advancement in training methods for large-scale reasoning models, shifting from coarse-grained sequence optimization to fine-grained token regulation, improving accuracy, reasoning quality, and efficiency. As the application of reasoning models expands, ERPO lays a technical foundation for next-generation training.