CoRD: Multi-Teacher Collaborative Stepwise Decoding for Distilling Long Chain-of-Thought Reasoning

CoRD (Collaborative Reasoning Distillation) is an innovative framework for distilling Long Chain-of-Thought (Long-CoT) reasoning. Through multi-teacher collaborative stepwise decoding, combined with perplexity scoring and beam search, it addresses issues in existing distillation methods such as blindness, lack of dynamic exploration, and missed complementary reasoning. It reduces redundant sampling while maintaining reasoning quality, enabling student models to achieve performance close to that of teacher models.

Large Reasoning Models (LRMs) excel in tasks like mathematical proof and code debugging with their Long-CoT capabilities, but their computational overhead is enormous. Knowledge distillation is key to transferring these capabilities to small models, yet existing methods have limitations:

1. Blindness in post-hoc filtering: Teachers generate trajectories independently before filtering, ignoring the collaborative potential of heterogeneous teachers;
2. Lack of dynamic exploration: Fixed sampling leads to redundant generation, and paths that deviate early still need to be completed;
3. Missed complementary reasoning: Unable to leverage the strengths of different teachers in different reasoning steps.

The core of CoRD is the multi-teacher collaborative stepwise synthesis mechanism:

• Step-level decision-making: Each reasoning step has multiple teachers propose candidate next steps;
• Perplexity-guided scoring: Use low perplexity (high model confidence) to filter high-quality steps;
• Beam search optimization: Retain the most promising reasoning hypotheses to balance exploration and efficiency;
• Heterogeneous teacher collaboration: Dynamically leverage the expertise of different teachers (e.g., mathematical symbol manipulation, logical deduction);
• Diverse hypothesis retention: Avoid premature convergence to suboptimal solutions.

Experiments validate CoRD's advantages:

• Reasoning data quality: Stepwise collaboration reduces error accumulation, with quality higher than post-hoc filtering;
• Student model performance: After training, student models approach teacher-level performance on multiple reasoning benchmarks, using fewer structured supervision signals;
• Efficiency: Stepwise filtering reduces invalid generation, and beam search pruning avoids exponential expansion without significant additional overhead;
• Generalization ability: Robust performance on out-of-domain tasks and open-ended questions.

Technical implementation details:

• Perplexity calculation: The formula is PPL = exp(-1/N * Σ log P(w_i | w_{<i})), used to evaluate the quality of individual reasoning steps;
• Beam search configuration: Supports flexible parameters like beam width, diversity penalty, and length normalization;
• Teacher model selection: 2-4 heterogeneous models (differences in architecture, training data, scale) achieve optimal cost-effectiveness.

Significance:

• Reduce deployment costs, enabling high-quality reasoning in resource-constrained environments;
• Promote an ecosystem of heterogeneous model collaboration;
• Improve sampling efficiency in data-scarce domains.

Limitations:

• Performance is limited by the upper bound of teacher model capabilities;
• The collaboration mechanism has additional computational overhead;
• Some tasks are not suitable for stepwise decomposition.

Future Directions:

• Adaptive teacher selection;
• Reinforcement learning optimization for beam search;
• Multimodal expansion (visual reasoning, code generation).

The research team has open-sourced the dataset and model on GitHub to facilitate result verification, community improvements, and technology dissemination.

Conclusion: CoRD is an important advancement in the field of knowledge distillation. Through multi-teacher collaboration and stepwise synthesis mechanisms, it bridges cutting-edge research and practical applications, providing a feasible path for the practical deployment of Long-CoT reasoning. It is worth in-depth exploration by researchers and engineers.