StableOPD: Addressing Length Inflation in Online Policy Distillation for Large Models

The research team reveals the length inflation and truncation collapse issues in Online Policy Distillation (OPD) training, proposes the StableOPD framework which combines reference divergence constraints and mixed rollout distillation, effectively improving training stability and achieving an average performance increase of 7.2% across multiple datasets.

The scaling of Large Language Models (LLMs) brings improved capabilities, but also increases deployment costs and inference latency, spurring the development of model distillation techniques. As an emerging paradigm, Online Policy Distillation (OPD) allows student models to train using their own generated responses, which theoretically enables learning the distribution encountered in practice, but in reality faces issues like training instability and collapse.

The study first reveals the length inflation phenomenon in OPD training: student model responses suddenly become longer, filled with repetition and redundancy; due to sequence length limits, overly long responses are truncated, leading to training data being dominated by truncated trajectories (truncation collapse), which is closely related to repetition saturation and exacerbates training instability.

The OPD objective function implicitly prefers long and repetitive responses (more overlapping opportunities, stable gradients), forming a feedback loop: long responses → high rewards → reinforcement of long response strategies → even longer responses, eventually leading to out-of-control length; some samples easily trigger inflation, distorting the training data distribution.

The StableOPD framework includes: 1. Reference Divergence Constraint: Limits the KL divergence between the student output distribution and the reference model (initial student or baseline model) to prevent excessive policy drift; 2. Mixed Rollout Distillation: Uses responses from student online rollouts, teacher outputs, manual annotations, etc., simultaneously to increase data diversity and smooth reward signals.

Validated on mathematical reasoning datasets such as GSM8K and MATH: 1. Prevents truncation collapse, maintains reasonable response lengths, and smooth training curves; 2. Average performance improvement of 7.2%; 3. Reduction of repeated n-gram ratio by approximately 40%; 4. Effective generalization across models from 7B to 70B parameters.

StableOPD brings the following insights: 1. Monitoring changes in response length is a key indicator of training health; 2. When training with model self-generated data, we need to be alert to distribution shifts and require constraint mechanisms; 3. Reward design needs to consider feedback loops and incentive distortions; 4. Mixing multiple signal sources can obtain more robust training signals.

StableOPD currently focuses mainly on mathematical reasoning tasks; its effectiveness in other domains needs to be verified; the choice of reference model affects performance, so automatic selection/dynamic adjustment needs to be explored; future research can focus on content-based dynamic length control or explicit length optimization objectives.