A Survey of Label-Free Reinforcement Learning: Cutting-Edge Exploration and Evaluation Reflections in the RLVR Field

This article surveys RLVR (Reinforcement Learning with Verifiable Rewards), a cutting-edge direction in the field of label-free reinforcement learning, exploring its progress in enhancing the reasoning capabilities of language models and reflecting on the issues in evaluation methods. The Label-Free-RLVR project is a community-maintained resource repository that compiles the latest research papers in this field, while reminding researchers to pay attention to potential problems in evaluation methods.

Traditional machine learning relies heavily on labeled data: supervised learning requires a large number of manually labeled samples, while reinforcement learning reduces dependence on labels but still needs carefully designed reward functions. In complex tasks such as mathematical reasoning and code generation, designing accurate reward signals is a huge challenge. In recent years, RLVR has emerged as a new paradigm, whose core is to use automatically verifiable feedback as reward signals without manual labeling or complex reward models, opening up a new path for improving the capabilities of large language models.

The key innovation of RLVR lies in the source of rewards: unlike RLHF, which relies on human preference annotations, RLVR uses the verifiability of the task itself (such as computational verification of mathematical answers, passing test cases for code). Its advantages include:

• Scalability: Automatic verification requires no human effort, allowing infinite expansion of training data;
• Objectivity: Clear verification standards avoid subjective biases in human annotations;
• Immediacy: Verification is performed in real time, supporting online learning. The training process is: model generates answers → system automatically verifies → calculates rewards based on results → updates the model using RL algorithms, and iteratively improves.

The Label-Free-RLVR project collects important papers in this field, with representative work including research on "Optimizing Reasoning Preferences Using Pseudo-Feedback", which demonstrates the potential of achieving effective learning without relying on external annotations. Research trends show that large language models are shifting from "imitating human examples" to "trial-and-error self-improvement" (similar to human learning). In structured tasks such as mathematical reasoning, logical puzzles, and code generation, RLVR methods perform prominently; models not only generate correct answers but also develop reasoning strategies and steps.

The Label-Free-RLVR project reminds that the improvements in many RLVR papers may be misleading. The main issues include:

• Baseline Underestimation: When comparing performance before and after RL, improper evaluation methods for the baseline model (such as different prompts, decoding parameters) can exaggerate RL improvements;
• Overfitting Risk: Repeated use of the validation set for training and tuning may lead the model to target specific validation sets rather than generalize;
• Verifiability Limitations: RLVR is difficult to apply to open-ended, creative, and subjectively evaluated tasks.

Based on evaluation reflections, the project puts forward the following suggestions:

1. Strict Baseline Evaluation: Ensure the baseline model is fully optimized, using the same prompts, decoding parameters, and evaluation protocols;
2. Separate Validation Sets: Distinguish between validation sets used for training feedback and test sets for final evaluation to prevent overfitting;
3. Multi-Dimensional Evaluation: Focus on reasoning processes, error types, generalization capabilities, etc., to avoid single metrics masking problems;
4. Ensure Reproducibility: Publicize code, data, and evaluation scripts to facilitate others to verify results.

The significance of RLVR lies in:

• Paradigm Shift: From imitation learning to autonomous exploration, which is closer to the formation mechanism of human intelligence;
• Applicability in Data-Scarce Fields: No need for high-quality annotations, suitable for scenarios such as professional knowledge and minority languages. Future directions include:
• Establishing standardized evaluation protocols to reduce biases;
• Extending to tasks with weak verifiability;
• Combining with technologies such as supervised fine-tuning and model distillation;
• Deepening theoretical understanding of the effectiveness and limits of RLVR. The Label-Free-RLVR project will continue to track these progressions.