REFT: Exploring Efficient Reinforcement Learning for Reasoning Models via First-Token Diversification

Title: REFT: Exploring Efficient Reinforcement Learning for Reasoning Models via First-Token Diversification

Core观点: This paper proposes the REFT method, addressing the sampling diversity bottleneck in Reinforcement Learning with Verifiable Rewards (RLVR). By introducing diversified sampling at the first token position after the reasoning prompt, it significantly enhances sampling diversity in a lightweight manner, outperforming DAPO and GRPO baselines across multiple models (0.5B-7B) and difficulty settings.

Source Information: Original authors: arXiv authors; Source platform: arXiv; Original title: Where Rollouts Begin: Low-Load, High-Leverage First-Token Diversification for RLVR; Link: http://arxiv.org/abs/2605.28295v1; Publication time: 2026-05-27T10:46:01Z.

Research Background: RLVR is an important paradigm for training reasoning models. It does not require pre-annotated reasoning trajectories, relying on grouped sampling to expose the policy model to various alternative reasoning paths, which are then scored by a verifier. However, RLVR faces the core challenge of insufficient sampling diversity—overly similar paths limit learning effectiveness. Existing methods (temperature adjustment, prefix sampling, etc.) have issues such as high computational overhead or limited diversity improvement.

Core Mechanism of REFT:

1. Uniform first-token sampling: Select uniformly from the model's top-N candidate first tokens, forcing exploration of low-probability but potentially effective starting points;
2. Balanced allocation: Ensure each candidate first token gets the same sampling opportunity;
3. Zero-intrusion design: Only modify the first-token sampling method; other components (temperature, verifier, etc.) remain unchanged, making it easy to integrate.

Comparison with Existing Methods:

Method Type	Computational Overhead	Implementation Complexity	Impact on Correctness
Temperature Adjustment	Medium	Low	May reduce correctness
Prefix Sampling	High	Medium	Requires additional storage
Sampling Selection	High	High	Introduces additional bias
REFT	Low	Extremely low	No impact

Experimental Setup: Covers four models from 0.5B to 7B, and three difficulty levels (basic/medium/high-difficulty reasoning problems). Performance Improvement: Pass@1/8/64 all outperform DAPO and GRPO baselines, indicating that first-token diversification enhances exploration effectiveness and the probability of correct answers. Efficiency Analysis: Extremely low computational overhead, no additional storage required, and stable training.

Technical Insights:

1. Value of key positions: The first token determines the direction of the reasoning chain; targeted diversification is more efficient than uniform resource allocation;
2. Decoupling diversity and correctness: Increasing diversity does not sacrifice accuracy;
3. Effectiveness of simple methods: Only modifying the first-token sampling strategy, no complex architecture needed, and the effect is significant.

Application Prospects:

• Plug-and-play: Easily integrated into existing RLVR workflows;
• Synergistic effect: Combine with other diversity technologies;
• Domain expansion: Extend to sequence tasks such as code generation and text creation; Expansion Suggestions: Explore adaptive adjustment strategies for first-token candidate sets.

Conclusion: REFT achieves a significant improvement in RLVR sampling diversity with extremely low overhead, making it a simple, efficient, zero-intrusion enhancement solution. Future Research Directions:

1. In-depth theoretical analysis of the importance of the first-token position and candidate set selection;
2. Validate effectiveness on non-mathematical reasoning tasks;
3. Combine with advanced RL algorithms such as PPO/DPO;
4. Dynamically adjust the intensity of first-token diversification.