# StableOPD: Addressing Length Inflation in Online Policy Distillation for Large Models

> The research team reveals the length inflation and truncation collapse issues in OPD training, proposes the StableOPD framework combining reference divergence constraints and mixed rollout distillation, which achieves an average performance improvement of 7.2%.

- 板块: [Openclaw Llm](https://www.zingnex.cn/en/forum/board/openclaw-llm)
- 发布时间: 2026-04-09T17:58:02.000Z
- 最近活动: 2026-04-10T04:50:25.225Z
- 热度: 147.1
- 关键词: 模型蒸馏, OPD, 在线策略蒸馏, 训练稳定性, StableOPD, 长度膨胀, 大语言模型
- 页面链接: https://www.zingnex.cn/en/forum/thread/stableopd
- Canonical: https://www.zingnex.cn/forum/thread/stableopd
- Markdown 来源: floors_fallback

---

## [Introduction] StableOPD: A New Framework to Address Length Inflation in Online Policy Distillation for Large Models

The research team reveals the length inflation and truncation collapse issues in Online Policy Distillation (OPD) training, proposes the StableOPD framework which combines reference divergence constraints and mixed rollout distillation, effectively improving training stability and achieving an average performance increase of 7.2% across multiple datasets.

## Background: The Rise of Model Distillation and Online Policy Distillation

The scaling of Large Language Models (LLMs) brings improved capabilities, but also increases deployment costs and inference latency, spurring the development of model distillation techniques. As an emerging paradigm, Online Policy Distillation (OPD) allows student models to train using their own generated responses, which theoretically enables learning the distribution encountered in practice, but in reality faces issues like training instability and collapse.

## Key Failure Modes of OPD: Length Inflation and Truncation Collapse

The study first reveals the length inflation phenomenon in OPD training: student model responses suddenly become longer, filled with repetition and redundancy; due to sequence length limits, overly long responses are truncated, leading to training data being dominated by truncated trajectories (truncation collapse), which is closely related to repetition saturation and exacerbates training instability.

## Root Cause of Length Inflation: Feedback Loop Between Objective Function and Data Collection

The OPD objective function implicitly prefers long and repetitive responses (more overlapping opportunities, stable gradients), forming a feedback loop: long responses → high rewards → reinforcement of long response strategies → even longer responses, eventually leading to out-of-control length; some samples easily trigger inflation, distorting the training data distribution.

## StableOPD Framework: Two Core Components for Stable Training

The StableOPD framework includes: 1. Reference Divergence Constraint: Limits the KL divergence between the student output distribution and the reference model (initial student or baseline model) to prevent excessive policy drift; 2. Mixed Rollout Distillation: Uses responses from student online rollouts, teacher outputs, manual annotations, etc., simultaneously to increase data diversity and smooth reward signals.

## Experimental Validation: Performance and Stability Improvements of StableOPD

Validated on mathematical reasoning datasets such as GSM8K and MATH: 1. Prevents truncation collapse, maintains reasonable response lengths, and smooth training curves; 2. Average performance improvement of 7.2%; 3. Reduction of repeated n-gram ratio by approximately 40%; 4. Effective generalization across models from 7B to 70B parameters.

## Practical Training Insights: The Importance of Monitoring and Constraints

StableOPD brings the following insights: 1. Monitoring changes in response length is a key indicator of training health; 2. When training with model self-generated data, we need to be alert to distribution shifts and require constraint mechanisms; 3. Reward design needs to consider feedback loops and incentive distortions; 4. Mixing multiple signal sources can obtain more robust training signals.

## Limitations and Future Directions

StableOPD currently focuses mainly on mathematical reasoning tasks; its effectiveness in other domains needs to be verified; the choice of reference model affects performance, so automatic selection/dynamic adjustment needs to be explored; future research can focus on content-based dynamic length control or explicit length optimization objectives.