RREDCoT: A Fine-Grained Reward Redistribution Mechanism for Reasoning Models

Introduction: RREDCoT—A New Approach to Solving Reward Allocation Challenges in Reasoning Models

RREDCoT proposes a fine-grained reward redistribution method for Chain-of-Thought (CoT) reasoning trajectories. Its core lies in using the model's own capabilities to approximate optimal reward allocation, addressing the delayed reward and high variance issues of traditional GRPO algorithms in long reasoning chains. This method improves accuracy, training stability, and reduces computational costs in tasks like mathematical reasoning and code generation, providing an effective framework for reasoning model training.

Research Background: Reward Dilemmas in Reasoning Model Training and Limitations of Existing Solutions

Challenges of Delayed Rewards

Long reasoning chains generated by reasoning models rely only on binary rewards from the final answer, leading to credit assignment difficulties (inability to distinguish between effective and ineffective steps), high variance (unstable training with Monte Carlo methods like GRPO), and high computational overhead for long contexts.

Limitations of Existing Solutions

• Monte Carlo Sampling: Unbiased but with extremely high computational cost, making it hard to apply to long chains.
• Attribution Techniques: Efficient but results are mostly correlational, making it difficult to handle long-range dependencies.

Core Method: RREDCoT's Fine-Grained Reward Redistribution Mechanism

Core Idea

Use the model's own output to approximate optimal reward allocation without additional sampling.

Key Components

1. Chain-of-Thought Segmentation: Divide into segments based on semantic completeness, granularity balance, and structure awareness (e.g., fixed length, semantic boundaries, adaptive segmentation).
2. State Value Estimation: Estimate segment values via bootstrapping (model prediction probability), iterative refinement, and variance control (baseline).
3. Reward Redistribution: Contribution weighting, error penalty, and smoothing.
4. Integration with GRPO: Plug-and-play compatibility, integrating segment rewards in group sampling, reward calculation, and policy update phases.

Experimental Validation: Performance Advantages of RREDCoT

Comparison Methods

Original GRPO, MC-GRPO, Attention Attribution, Gradient Attribution.

Evaluation Metrics

Task accuracy, training stability, sample efficiency, reasoning quality.

Key Results

• Accuracy: Outperforms original GRPO in math/code tasks, close to MC-GRPO.
• Stability: Significantly reduces reward variance, with smoother learning curves.
• Efficiency: Training time reduced by over 60% compared to MC-GRPO.
• Fine-grained: Accurately identifies key reasoning steps.

Practical Recommendations: Application Guide for Model Developers and Researchers

Model Developers

1. Segmentation Granularity: Start with semantic boundaries and adjust as needed.
2. Hyperparameters: Tune value estimation weights and regularization coefficients.
3. Monitoring: Pay attention to both final accuracy and the rationality of segment rewards.

Researchers

1. Interpretability: Use reward allocation to analyze model behavior.
2. Error Diagnosis: Locate weak points via negative rewards.
3. Data Filtering: Use state value estimation to filter high-quality samples.

Conclusion and Outlook: Value of RREDCoT and Future Directions

Conclusion

RREDCoT achieves fine-grained reward allocation through the model's own capabilities, improving training stability and performance, and providing an effective framework for reasoning model training.

Limitations

• Relies on segmentation quality; automatic segmentation needs improvement.
• Experiments are focused on math/code tasks; cross-domain validation is needed.
• Theoretical analysis needs to be improved.

Future Directions

End-to-end segmentation learning, hierarchical reward allocation, cross-task transfer, integration with RLHF.