In-depth Analysis of Policy Distillation for Large Language Models: Phenomena, Mechanisms, and Practical Guide

This article focuses on the Off-Policy Distillation (OPD) technique in the post-training of large language models. The Tsinghua University research team systematically reveals two key conditions for its success—reasoning mode compatibility and the teacher model providing new capabilities—and proposes practical improvement methods such as off-policy cold start and teacher-aligned prompt selection. It also discusses the hidden costs of OPD and future research directions.

In recent years, large language models (LLMs) have entered the post-training phase, and Off-Policy Distillation (OPD) has become one of the core technologies. Unlike traditional supervised fine-tuning, OPD allows student models to interact with teacher models in real time to obtain rich learning signals. However, despite its significant practical effectiveness, it lacks a systematic theoretical explanation, which the Tsinghua team's research fills.

Policy distillation is a special knowledge distillation method whose core is strategic data generation: the student model's output serves as training samples, and the teacher model provides scoring feedback. Its advantages include dynamic adaptation (the student's exploration space determines the data distribution), dense rewards (each token receives feedback), and capability transfer (learning the teacher's context-specific behaviors). However, its dynamic nature brings complexity to the variation in effectiveness.

1. Reasoning Mode Compatibility: The student and teacher models need to adopt similar reasoning paths and representation methods. Models from the same family (e.g., Qwen or Llama series) have higher compatibility, and the effect of a small-scale same-family teacher may be comparable to that of a cross-family large model; 2. Teacher Provides New Capabilities: The teacher must demonstrate reasoning skills, knowledge boundaries, or solutions that the student has not mastered. Repeating only known content cannot bring substantial improvement.

Successful OPD exhibits three key features: 1. Progressive Alignment: The student gradually aligns with the teacher's high-probability tokens; 2. Small Core Token Set: 97%-99% of the probability mass is concentrated in a small shared token set; 3. Importance of State Access: The context generated by the student determines what it can learn from the teacher.

1. Off-Policy Cold Start: Introduce external data sources or strong generators in the early training stage to expand the output space, then switch to standard OPD after the student gains basic exploration capabilities; 2. Teacher-Aligned Prompt Selection: Prioritize prompts where the teacher can demonstrate obvious advantages and provide new insights to improve learning efficiency.

Implications: Prioritize same-family teachers, verify teacher novelty, monitor core token alignment, and combine off-policy data with OPD; Hidden Costs: Long-range tasks face issues of credit assignment, exploration-exploitation trade-off, and computational overhead; Future Directions: Explore OPD expansion in complex long-range tasks and design more efficient and scalable distillation strategies.