Unveiling the Truth of Policy Distillation: When It Works, When It Harms, and Why

The core research in this article proposes a training-free token-wise diagnostic framework. Through detailed analysis, it finds: policy distillation is more helpful on incorrect reasoning paths, while teacher signals may introduce noise on correct paths. This framework can quantify the alignment between distillation configurations and ideal gradients, providing key guidance for training reasoning models, explaining the unstable effects of distillation, and offering optimization directions.

Policy distillation is a common technique for training reasoning models. It generates complete reasoning chains and performs supervised learning based on teacher feedback, playing an important role in training models like DeepSeek-R1 and OpenAI o1. However, its effects are unstable in practice—sometimes improving capabilities, sometimes being mediocre or even harmful. Traditional methods require a large number of expensive experiments, and aggregated metrics mask token-wise dynamic changes, lacking fine-grained analysis tools.

The proposed diagnostic framework includes three key innovations:

1. Definition of Ideal Gradient: The parameter update direction that maximizes the student model's success probability, representing the theoretical upper limit of supervision signals;
2. Targeted Rollout Algorithm: Efficiently estimates the ideal gradient for long reasoning chains through importance sampling, Monte Carlo Tree Search (MCTS), and variance reduction techniques;
3. Gradient Alignment Score: Quantifies the cosine similarity between actual distillation gradients and ideal gradients to measure configuration effectiveness. These technologies ensure reliable gradient estimation at a reasonable cost.

Experiments reveal a counterintuitive phenomenon: policy distillation has higher alignment with ideal gradients on incorrect reasoning paths, and lower alignment on correct paths. The reason is that when the student is correct, teacher signals may introduce bias; when incorrect, teacher signals provide effective correction. Additionally, the optimal distillation configuration depends on student capabilities and task characteristics—there is no one-size-fits-all solution.

The study provides guidance for training practices including:

• Dynamic Distillation: Adjust based on the student's real-time performance—strengthen in the early stage and weaken when converging;
• Integration with Curriculum Learning: Transition from simple problems (reducing interference) to difficult problems (increasing guidance);
• Teacher Selection: Teachers with similar capabilities may better match the student's error patterns;
• Real-time Monitoring: Integrate the framework to monitor gradient alignment scores, triggering configuration adjustments or early stopping.

The framework has limitations: ideal gradient estimation has approximation errors (may be inaccurate for extremely long chains), and it currently applies to autoregressive models—other architectures need verification. Future directions include developing an online version to apply diagnostic results in real time, extending to multimodal reasoning, and exploring interaction effects with technologies like reinforcement learning.

This study provides a theoretical perspective and practical tools for policy distillation, revealing that effective teaching requires providing guidance when students need it. These findings will help researchers and engineers design more intelligent and efficient training strategies for reasoning models.