GRPO-VPS: Verifiable Process Supervision Enhances LLM Reasoning Efficiency

This article proposes the GRPO-VPS (Verifiable Process Supervision) method, which achieves fine-grained process supervision by detecting belief changes during the model's reasoning process. Without requiring additional models or Monte Carlo sampling, this method achieves a 2.6% accuracy improvement and a 13.7% reduction in reasoning length on mathematical reasoning tasks, balancing reasoning effectiveness and efficiency.

Dilemmas in Reasoning Training

Traditional Supervised Fine-tuning (SFT) relies on manual annotation of reasoning processes, which is costly and difficult to scale. The Reinforcement Learning with Verifiable Rewards (RLVR) paradigm provides signals by verifying the final answer without process annotation.

Pain Points of GRPO

Group Relative Policy Optimization (GRPO) eliminates the dependency on critic models, but its trajectory-level feedback mechanism leads to coarse-grained credit assignment:

1. Unable to identify effective reasoning strategies, making it hard to locate error steps;
2. Models tend to overthink, generating lengthy reasoning chains that reduce efficiency.

Core Insight: Belief Detection

Judge the reasoning direction by measuring changes in the model's conditional probability of the correct answer during reasoning: rising belief → positive contribution, falling → error/deviation, stagnant → redundant.

Technical Implementation

1. Reasoning Segmentation: Divide steps based on natural language or logical structure;
2. Belief Detection: Calculate the model's conditional probability of the correct answer at segment boundaries;
3. Progress Measurement: Evaluate paragraph contributions by comparing belief changes between adjacent segments.

Advantages

• Model-agnostic: Directly uses the main model's probability estimation without additional parameters;
• Zero extra cost: No Monte Carlo sampling needed, reducing computational overhead;
• High interpretability: Paragraph-level progress facilitates understanding and debugging.

Training Process

Integrate paragraph-level progress into GRPO training: assign higher advantage estimates to paragraphs with positive progress, penalize those with falling beliefs, and encourage conciseness for redundant paragraphs to improve sample efficiency.

Experimental Results

• Mathematical Reasoning: 2.6% accuracy improvement, 13.7% reduction in reasoning length;
• General Domain: 2.4% accuracy improvement, 4% reduction in reasoning length;
• Cross-model Consistency: Stable improvements across multiple models.

Method Comparison

Method	Process Supervision	Additional Model	Computational Cost	Main Limitation
GRPO	No	No	Low	Coarse-grained feedback
PRM-based	Yes	Needs PRM	Medium	High PRM training cost
MCTS/Tree	Yes	No	High	High Monte Carlo sampling overhead
GRPO-VPS	Yes	No	Low	Need to design segmentation strategy

1. Reasoning Efficiency Optimization: Suppress redundant reasoning and reduce computational costs;
2. Error Diagnosis: Visualize reasoning processes and locate error-prone links;
3. Human-Machine Collaboration: Intervene in paragraphs where the model lacks confidence;
4. Educational Applications: Identify students' reasoning misconceptions and provide targeted feedback.

1. Segmentation Strategy Dependency: Reasoning with unclear structure is difficult to segment reasonably;
2. Belief Calibration Issue: Model probability estimates may have calibration biases;
3. Complex Reasoning Challenges: Belief changes are hard to capture quality in multi-hop/creative reasoning;
4. Answer Leakage Distinction: Need to distinguish between pattern matching and real reasoning progress.

GRPO-VPS achieves fine-grained process supervision without additional annotations through the belief detection mechanism, providing new ideas for the development of the RLVR paradigm. It improves both reasoning accuracy and efficiency, and has important value for the application of LLMs in complex reasoning fields such as mathematics and science.