# Unveiling the Truth of Policy Distillation: When It Works, When It Harms, and Why

> The study proposes a training-free diagnostic framework. Through token-wise analysis, it finds that policy distillation is more helpful on incorrect reasoning paths, while teacher signals may introduce noise on correct paths.

- 板块: [Openclaw Llm](https://www.zingnex.cn/en/forum/board/openclaw-llm)
- 发布时间: 2026-05-11T17:33:28.000Z
- 最近活动: 2026-05-12T06:21:32.539Z
- 热度: 138.2
- 关键词: 策略蒸馏, 推理模型, 教师模型, 梯度对齐, 训练诊断, 思维链, 自蒸馏, 模型训练
- 页面链接: https://www.zingnex.cn/en/forum/thread/llm-arxiv-2605-10889v1
- Canonical: https://www.zingnex.cn/forum/thread/llm-arxiv-2605-10889v1
- Markdown 来源: floors_fallback

---

## [Overview] Unveiling the Truth of Policy Distillation: When It Works, When It Harms, and Why

The core research in this article proposes a training-free token-wise diagnostic framework. Through detailed analysis, it finds: policy distillation is more helpful on incorrect reasoning paths, while teacher signals may introduce noise on correct paths. This framework can quantify the alignment between distillation configurations and ideal gradients, providing key guidance for training reasoning models, explaining the unstable effects of distillation, and offering optimization directions.

## Background: The Duality of Policy Distillation and Limitations of Traditional Methods

Policy distillation is a common technique for training reasoning models. It generates complete reasoning chains and performs supervised learning based on teacher feedback, playing an important role in training models like DeepSeek-R1 and OpenAI o1. However, its effects are unstable in practice—sometimes improving capabilities, sometimes being mediocre or even harmful. Traditional methods require a large number of expensive experiments, and aggregated metrics mask token-wise dynamic changes, lacking fine-grained analysis tools.

## Methodology: Token-wise Diagnostic Framework and Core Technologies

The proposed diagnostic framework includes three key innovations:
1. **Definition of Ideal Gradient**: The parameter update direction that maximizes the student model's success probability, representing the theoretical upper limit of supervision signals;
2. **Targeted Rollout Algorithm**: Efficiently estimates the ideal gradient for long reasoning chains through importance sampling, Monte Carlo Tree Search (MCTS), and variance reduction techniques;
3. **Gradient Alignment Score**: Quantifies the cosine similarity between actual distillation gradients and ideal gradients to measure configuration effectiveness.
These technologies ensure reliable gradient estimation at a reasonable cost.

## Key Findings: Value of Incorrect Paths and Configuration Dependency

Experiments reveal a counterintuitive phenomenon: policy distillation has higher alignment with ideal gradients on incorrect reasoning paths, and lower alignment on correct paths. The reason is that when the student is correct, teacher signals may introduce bias; when incorrect, teacher signals provide effective correction. Additionally, the optimal distillation configuration depends on student capabilities and task characteristics—there is no one-size-fits-all solution.

## Practical Implications: Dynamic Optimization of Distillation Strategies

The study provides guidance for training practices including:
- **Dynamic Distillation**: Adjust based on the student's real-time performance—strengthen in the early stage and weaken when converging;
- **Integration with Curriculum Learning**: Transition from simple problems (reducing interference) to difficult problems (increasing guidance);
- **Teacher Selection**: Teachers with similar capabilities may better match the student's error patterns;
- **Real-time Monitoring**: Integrate the framework to monitor gradient alignment scores, triggering configuration adjustments or early stopping.

## Limitations and Future Research Directions

The framework has limitations: ideal gradient estimation has approximation errors (may be inaccurate for extremely long chains), and it currently applies to autoregressive models—other architectures need verification. Future directions include developing an online version to apply diagnostic results in real time, extending to multimodal reasoning, and exploring interaction effects with technologies like reinforcement learning.

## Conclusion: A New Perspective on Intelligent Training Strategies

This study provides a theoretical perspective and practical tools for policy distillation, revealing that effective teaching requires providing guidance when students need it. These findings will help researchers and engineers design more intelligent and efficient training strategies for reasoning models.