SePT: A Reward-Model-Free Self-Training Reasoning Framework for LLMs

SePT (Self-Training with Process Rewards) is a novel reward-model-free self-training method designed to enable large language models (LLMs) to continuously improve their reasoning capabilities through self-generated process reward signals, opening up a new path to reduce RLHF training costs. Its core idea is "process as reward": by generating candidate reasoning paths, self-evaluating the quality of the process, and bootstrapping to learn effective strategies, it addresses the bottlenecks of traditional RLHF such as reliance on expensive annotated data and poor generalization of reward models. It has shown excellent experimental performance and significant application value.

Current mainstream LLM reasoning enhancement methods rely on the RLHF paradigm: collecting human preference data → training a reward model → reinforcement learning fine-tuning. However, there are three major bottlenecks: 1. High data cost (requiring a large amount of manually annotated preference comparison data); 2. Limited generalization ability of reward models (unstable on out-of-distribution data, prone to reward hacking); 3. Lack of autonomous improvement mechanism (tied to external evaluation systems). The SePT team proposes a solution where the model acts as its own teacher and learns to improve from its own generation process.

The core concept of SePT is "process as reward", focusing on the quality of each step in the reasoning process rather than just the final answer. The specific steps are: 1. Generate multiple candidate solution paths; 2. Evaluate the quality of each step through logical consistency, mathematical correctness, and semantic coherence (without requiring a pre-trained reward model); 3. Bootstrapping strategy: Identify high-quality reasoning patterns from its own multiple paths, learn effective strategies through contrastive learning, and achieve self-improvement without external supervision.

The technical components of SePT include: 1. Process decomposition module: Split complex reasoning tasks into evaluable atomic steps (e.g., mathematical formula transformation, code function calls); 2. Self-consistency evaluation: Use the model's own knowledge to verify the rationality of steps (e.g., mathematical substitution verification, logical counterexample checking); 3. Strategy optimization: Improved policy gradient method based on dynamic process quality scores; 4. Curriculum learning: Progressive training from simple to complex tasks to improve efficiency and the ability to handle complex tasks.

SePT has shown excellent performance in multiple reasoning benchmark tests: 1. Significantly outperforms baselines (without external reward models) on the GSM8K mathematical reasoning dataset; 2. Demonstrates cross-task generalization ability on the MATH competition-level dataset (stable performance on new question types); 3. Alleviates the model collapse problem of traditional self-training, with high training stability; 4. Without requiring a reward model, memory usage and computational overhead are significantly reduced, and computational efficiency is improved.

The application significance of SePT includes: 1. Reducing reliance on manual annotation and promoting the democratization of AI technology; 2. Supporting continuous learning (learning from new interactions and evolving dynamically in actual deployment); 3. Providing a new perspective for model interpretability (understanding the decision-making process by analyzing changes in step scores); 4. Inspiring metacognitive learning in the education field (learning to evaluate and improve thinking processes).

SePT has limitations: 1. Process evaluation relies on the capabilities of the base model (evaluation is unreliable when beyond its knowledge scope); 2. Limited applicable tasks (mainly for tasks with decomposable steps, such as mathematics and code); 3. Generating multiple paths still requires a large amount of computation. Future directions: Combine external knowledge bases/tools to enhance evaluation accuracy, expand to open creative tasks, explore efficient sampling and evaluation strategies, and complement RLHF to build a stronger training system.