VeriGate: Introducing a Stepwise Supervision Mechanism with Validator Gating to GRPO for Enhancing Large Model Reasoning Capabilities

VeriGate improves GRPO training through a validator gating mechanism. It enables process supervision when validator rewards are ineffective, converts PRM step scores into future cumulative rewards for fine-grained credit assignment, significantly reduces zero-gradient failures and reward gaming behaviors, and enhances large model reasoning capabilities.

GRPO trains reasoning models using result-based rewards from validators, but the supervision signals are overly sparse: when all sampled trajectories receive the same validator reward, the group relative advantage collapses to zero, leading to learning stagnation; result-only rewards fail to provide step-level credit assignment, limiting the model's exploration ability.

1. Validator Gating Mechanism: Continue using validator rewards when they can rank trajectories; activate process supervision when they degrade. 2. Future Cumulative Reward Conversion: Convert PRM step scores into future cumulative rewards, considering the step's impact on subsequent steps. 3. Group-Normalized Token-Level Advantage: Convert rewards into group-normalized token-level advantages to restore gradient signals and be more robust against reward gaming.

Trained on the MATH dataset using Qwen2.5-Instruct models with 1.5B and 7B parameters, evaluated on six reasoning benchmarks: the 1.5B model achieved an average accuracy improvement of about 20%, and the 7B model about 12%; it also reduced zero-gradient failures and reward gaming, improving reasoning quality.

Technical Significance: Provides a more refined supervision mechanism, solves the problems of sparse supervision and coarse credit assignment, and reduces reward gaming. Application Prospects: Can be used in multi-step tasks such as mathematical reasoning, code generation, and logical reasoning; it will become more important as large models are increasingly applied to complex tasks.

Training reasoning models requires attention to intermediate process supervision. Combining validators with process reward models and fine-grained credit assignment are key to enhancing reasoning capabilities, providing references for future reasoning model training method design.