Prune-OPD: An Efficient and Reliable Policy Distillation Method for Long-Range Reasoning

This paper proposes the Prune-OPD framework, which addresses the prefix drift problem in policy distillation for long-range reasoning tasks. By dynamically monitoring the local consistency between student and teacher predictions, it reduces training time by 37.6%-68.0% while maintaining or even improving model performance, providing an efficient and reliable strategy for training long-range reasoning models.

Policy Distillation (OPD) is an important technique to enhance the reasoning ability of large language models, but it faces the "prefix drift" problem when extended to long-range reasoning: the reasoning prefix generated by the student deviates from the teacher's thinking, causing the dense rewards provided by the teacher to lose local usability, which both reduces reward quality and leads to computational waste.

The Prune-OPD framework includes two key components: drift detection and dynamic truncation. It monitors the local compatibility between student and teacher predictions through top-k overlap; when drift is detected, it downweights unreliable rewards and initiates dynamic rollout truncation. It also adopts an adaptive strategy: when compatibility is high, long-context supervision is retained; when drift occurs, resources are truncated and reallocated to achieve optimal computational efficiency.

On mathematical reasoning benchmarks such as AMC, AIME, and HMMT, Prune-OPD reduces training time by 37.6%-68.0% compared to standard OPD, while maintaining or improving performance. It shows good generalization across different teacher-student model combinations, stably optimizing computational efficiency and performance.

The success of Prune-OPD shows that intelligently filtering teacher signals and adopting quality-aware learning strategies in knowledge distillation can significantly improve training efficiency while ensuring performance. This idea can be extended to the training of a wider range of AI systems, providing a reference for dynamic allocation of computational resources.

Prune-OPD has limitations: using top-k overlap as a drift detection metric may not be optimal, and the dynamic truncation strategy may be aggressive in extreme cases. Future research can explore more refined drift detection mechanisms and more conservative truncation strategies to balance efficiency and performance in a wider range of scenarios.