RLAD：面向大语言模型推理的强化感知知识蒸馏新方法

RLAD proposes an innovative knowledge distillation framework that effectively transfers the reasoning ability of teacher models during reinforcement learning training through selective imitation and Trust Region Ratio Distillation (TRRD) techniques. It solves the core problem of integrating knowledge distillation and reinforcement learning, enabling small models to not only learn how to reason but also understand why to reason that way.

Knowledge distillation (KD) and reinforcement learning (RL) are two important technical routes to enhance LLM capabilities. However, combining them for reasoning improvement faces fundamental difficulties: traditional offline KD cannot adapt to the evolving strategy distribution of student models in RL; KL divergence-based distillation overconstrains the student's exploration space and harms reasoning quality; pure RL wastes the valuable knowledge accumulated by teacher models. RLAD is proposed to address this fusion problem.

Traditional KD assumes all teacher outputs are worth learning, but this is not true in dynamic RL training. Selective imitation decides whether to use teacher guidance by evaluating three questions: Is the student's current rollout distribution aligned with the teacher's strategy? Will imitating the teacher improve the expected reward for this sample? Is the current state suitable for RL free exploration? Only when these conditions are met does the teacher's knowledge get introduced, avoiding negative effects of blind imitation.

Traditional KL divergence distillation constrains students to the neighborhood of the teacher's strategy, limiting exploration. TRRD uses a likelihood ratio-based objective function to balance exploration, exploitation, and imitation. Its core idea is to measure the innovation of the student's behavior by comparing the ratio of student and teacher strategies. When the ratio is within a reasonable range, the student can learn from the teacher while retaining exploration freedom; when it deviates too much, constraints are applied to prevent strategy collapse. Its mathematical form is similar to PPO's clipping objective but applied to distillation, maintaining stability without manual hyperparameter adjustment.

RLAD's training process includes three steps: 1. Trajectory collection: Fix the teacher model to generate high-quality reasoning trajectories (including full reasoning processes). 2. Selective evaluation: Evaluate each trajectory based on alignment threshold and advantage value threshold; only qualified ones enter distillation. 3. Joint optimization: The student model receives both RL reward signals and TRRD distillation signals; the two losses are weighted to form the final optimization target, allowing the student to learn teacher's reasoning patterns and discover new effective strategies through trial and error.

RLAD has three main advantages: 1. Sample efficiency improvement: Selective use of teacher knowledge avoids wasting resources on invalid samples, significantly improving sample efficiency. 2. Reasoning quality guarantee: Compared to pure RL, it retains the teacher's reasoning structure, ensuring readable and logical reasoning processes (critical for interpretable applications). 3. Model scale flexibility: It adapts to different model scales, from distilling千亿-parameter teachers to百亿-parameter students or smaller ones by adjusting hyperparameters.

RLAD was verified on multiple reasoning tasks (math, code generation, logical reasoning). Results show that RLAD-trained students outperform traditional distillation and pure RL in accuracy, with good generalization on out-of-distribution test sets. Ablation experiments confirm that removing either selective imitation or TRRD leads to performance degradation, proving their synergistic effect is key to RLAD's success.

RLAD's application prospects include: 1. Model compression and deployment: Reducing model size while maintaining reasoning ability to lower deployment costs. 2. Reasoning ability improvement: Efficient fine-tuning path for domain-specific reasoning (using general large teachers and task-focused students).3. Research inspiration: Inspiring fusion of other learning paradigms. Summary: RLAD solves the core problem of KD-RL integration via selective imitation and TRRD, improving student reasoning ability and providing a new methodology for efficient LLM training, which will play an important role in future model optimization.