OneShotTrainingExample: A One-Shot RLVR Selector Training Framework for Mathematical Reasoning Models

OneShotTrainingExample is a unified workspace that integrates GHPO/Open-R1 training code and one-shot RLVR selector experiments. It aims to efficiently enhance the performance of mathematical reasoning models through the one-shot RLVR selector, providing a complete training, evaluation, and analysis workflow. This addresses the issues of insufficient deep reasoning capabilities in traditional supervised fine-tuning (SFT) and the high resource consumption and complex tuning of reinforcement learning (RL) training.

Project Background

In the field of large language models, mathematical reasoning ability is a key indicator of model intelligence. Traditional supervised fine-tuning (SFT) lacks deep reasoning capabilities for complex mathematical problems; while reinforcement learning (RL) can guide models to learn reasoning strategies, it requires significant computational resources and complex hyperparameter tuning.

Core Objectives

The OneShotTrainingExample project provides a unified workspace, integrating GHPO and Open-R1 training code, focusing on efficiently improving the performance of mathematical reasoning models through the one-shot RLVR selector.

GHPO: Group Hindsight Policy Optimization

• Definition: Group Hindsight Policy Optimization, compared to traditional PPO, uses group sampling results to guide policy updates
• Advantages: More efficient use of computational resources, extracting training signals from multiple candidate answers (even if partially correct)
• Code Support: Includes complete training code, configuration files, and scripts, supporting fine-tuning of models like Qwen2.5-Math-7B, with hyperparameters managed via YAML

One-Shot RLVR Selector Mechanism

• Definition: A reinforcement learning paradigm based on verifiable rewards, training a selector to judge whether a candidate reasoning path leads to the correct answer
• Advantages: Reduces training difficulty (only evaluation instead of generation), improves sample efficiency (utilizes existing paths), and enhances interpretability (analyzes decision-making processes)
• Experiment Support: Provides a complete Jupyter Notebook experimental workflow, forming a closed loop from reasoning to selector experiments

The project's experiments follow a step-by-step principle, divided into multiple phases:

1. Phase 0: Establish basic reasoning capabilities, collect baseline data using pre-trained models
2. Phase 1: Improve reasoning tests, explore prompt strategies and reasoning path generation methods
3. Phase 4: Core selector experiments, train multi-variant selectors and compare performance on the validation set
4. Phase 5: Build a complete training loop, combining the selector with the base model to achieve end-to-end improvement

Each phase has a corresponding Notebook that records steps, parameters, and result visualization, facilitating reproduction and adjustment.

• Research_Findings.md: Summarizes key experimental findings, including quantitative results (accuracy improvement) and qualitative analysis (performance differences across problems of varying difficulty)
• presentation_script.md: Presentation script, suitable for academic conferences or team sharing
• RL-FinalPush.ipynb: Summary Notebook, integrating results from all phases, providing performance comparisons and conclusions

Documentation ensures research results are traceable and reproducible.

Tech Stack

• Training framework: PyTorch
• Model operations: Transformers library
• Experimental environment: Jupyter
• Dependency management: requirements.txt lists all Python packages

Deployment Recommendations

1. Requires GPU resources (especially for 7B parameter models)
2. Large model weights/checkpoints are not committed to Git
3. Execute Notebooks in the order specified in Documents/Execution-Step_Smit.txt

Application Scenarios

• Math education: Evaluate the quality of students' problem-solving processes
• Model development: Provide an efficient RL training path
• Academic research: Open up a new direction of validator design guiding generative models

Future Outlook

• Expand to code generation, logic puzzles, scientific question answering, and other fields
• Community contributions: Support more base models, optimize selector architecture, and apply to new tasks

The open-source nature of the project provides space for community improvements.