MCPO: Enhancing Large Model Reasoning Ability via Mastery Consolidation and Optimization

Aiming at the training signal issue of the GRPO algorithm on mastered prompts (accuracy close to 100%) and mostly correct prompts (50%-100% accuracy), this paper proposes the MCPO framework. Core innovations include hinge KL regularization (constraining policy drift for mastered prompts) and a weighting mechanism for mostly correct prompts. It achieves continuous improvement in pass@1 performance on mathematical reasoning benchmarks and unexpectedly enhances pass@k diversity.

Background: The Rise of RLVR and GRPO

Reinforcement Learning with Verifiable Rewards (RLVR) leverages automatic verification signals (such as mathematical correctness) to enhance large model reasoning ability without manual reward annotation. As a member of the RLVR family, GRPO calculates the advantage function by comparing the relative quality of multiple outputs under the same prompt, avoiding the overhead of training a separate critic model in traditional PPO and achieving efficient performance.

Core Issues of GRPO

Problem 1: Vanishing Training Signals for Mastered Prompts

When prompt accuracy is close to 100%, all sampled outputs are correct, and the relative advantage approaches zero, leading to no effective training signals, policy drift, and catastrophic forgetting.

Problem 2: Weight Decay for Mostly Correct Prompts

For prompts with accuracy between 50% and 100%, GRPO's query weight shrinks as accuracy increases. This reduces the model's optimization intensity during the phase from partial correctness to full mastery, weakening consolidation learning.

Key Innovations of MCPO

Innovation 1: Hinge KL Regularization

For mastered prompts, a hinge loss mechanism constrains drastic policy distribution changes—punishment is applied only when drift exceeds a threshold, preventing catastrophic forgetting while retaining beneficial exploration.

Innovation 2: Weighting Mechanism for Mostly Correct Prompts

Re-weighting mostly correct prompts ensures the model receives sufficient training signals when approaching mastery, enabling a smooth transition to full mastery and improving learning efficiency.

Experimental Results: Dual Improvement in Performance and Diversity

On three mathematical benchmarks—GSM8K (elementary school math), MATH (competition level), and OlympiadBench (Olympiad)—MCPO continuously improves pass@1 (single-sample accuracy).

Unexpected finding: pass@k (probability of at least one correct answer in k samples) is significantly enhanced, reflecting increased diversity in the solution space. This breaks traditional perceptions: consolidation learning not only does not limit exploration but also catalyzes diversity; a stable base policy provides a solid starting point for exploration.

Reasons for MCPO's Effectiveness

Stable Foundation Promotes Exploration

By preventing the forgetting of mastered knowledge, the model gains a stable and reliable foundation, allowing it to explore new areas more confidently without worrying about damaging existing knowledge—making exploration more efficient.

Optimized Resource Allocation

Re-weighting mostly correct prompts avoids wasting computation on mastered prompts, ensuring problems close to mastery receive sufficient attention, leading to a smoother and more efficient learning curve.

Implications and Future Directions

Implications for RLVR Practice

• Monitor the distribution of prompt mastery
• Implement special handling for mastered prompts (e.g., regularization)
• Dynamically adjust prompt weights to optimize learning

Limitations and Future

Current limitations: Experiments are focused on the math domain; the hinge KL threshold requires task-specific tuning; the effect on ultra-large-scale models remains untested. Future directions: Cross-domain verification (code generation, scientific reasoning); adaptive thresholds; combined strategies; theoretical analysis of the mathematical relationship between mastery and diversity.