AsymGRPO: Rethinking Exploration Mechanisms in RLVR—From Entropy Regularization to Bidirectional Entropy Modulation

This article introduces the AsymGRPO framework, which decomposes policy entropy into 'informational entropy' (beneficial uncertainty) and 'spurious entropy' (unhelpful noise) to enable differential modulation of positive and negative samples. This addresses the exploration limitation issue of large language models in Reinforcement Learning with Verifiable Rewards (RLVR), enhancing their reasoning ability and generalization performance.

In recent years, Reinforcement Learning with Verifiable Rewards (RLVR) has become a mainstream paradigm for enhancing the reasoning ability of Large Language Models (LLMs). It allows models to optimize their reasoning strategies through trial and error via automatically verified reward signals. However, this paradigm faces a fundamental bottleneck: the policy network quickly converges to a narrow solution space, falling into local optima, tending to repeat known paths, ignoring potentially better solutions, and limiting the discovery of novel approaches and generalization performance.

To alleviate exploration limitations, traditional methods use entropy regularization to encourage action diversity, but they have flaws in LLM scenarios: hyperparameter sensitivity (minor changes lead to unstable training or sudden performance drops), diminishing marginal returns (simply increasing entropy regularization yields limited improvements), and blindness (failing to distinguish between 'good' diversity and 'bad' noise).

Starting from Group Relative Policy Optimization (GRPO), the research proposes the decomposition of policy entropy into informational entropy (beneficial uncertainty that preserves diverse solutions) and spurious entropy (unhelpful noise that erodes reasoning). GRPO has an embedded implicit entropy refinement mechanism: it maintains informational entropy for positive samples (high-reward trajectories) and suppresses spurious entropy for negative samples (low-reward trajectories), but the modes of action are implicitly coupled.

The core innovation of the AsymGRPO framework is the explicit decoupling of entropy modulation for positive and negative samples: it proactively retains and enhances informational entropy for positive samples to encourage exploration diversity on successful paths; for negative samples, it actively suppresses spurious entropy to reduce ineffective attempts in wrong directions. This brings higher controllability (independent adjustment of intensity), better stability (reduced hyperparameter interference), and stronger compatibility (synergy with existing entropy regularization).

Evaluations on multiple benchmark tasks show: AsymGRPO significantly outperforms strong baseline methods; it demonstrates synergistic potential when combined with existing entropy regularization techniques; its sensitivity to hyperparameter changes is significantly reduced. This validates the effectiveness of the entropy refinement framework and provides guidance for RLVR practice: focus on refining the composition of entropy and distinguishing between beneficial and unhelpful uncertainty.

AsymGRPO reveals key insights: the quality of entropy is more important than quantity; positive and negative samples should be treated differently; explicitization of implicit mechanisms can improve performance. Future directions include extending to more complex reasoning tasks, exploring combinations with other regularization techniques, and deepening the mathematical characterization of informational entropy and spurious entropy.