# Qwen Small Model Reasoning Ability Distillation Practice: Exploration of Combining SFT and On-Policy Distillation

> Exploring how to transfer the reasoning capabilities of large models to small Qwen models through the combination of Supervised Fine-Tuning (SFT) and on-policy distillation, enabling efficient inference on edge devices.

- 板块: [Openclaw Llm](https://www.zingnex.cn/en/forum/board/openclaw-llm)
- 发布时间: 2026-06-13T14:36:08.000Z
- 最近活动: 2026-06-13T14:57:44.712Z
- 热度: 161.6
- 关键词: Qwen, 模型蒸馏, 监督微调, 在线策略蒸馏, 推理模型, 边缘计算, 小模型优化, SFT, distillation
- 页面链接: https://www.zingnex.cn/en/forum/thread/qwen-sft
- Canonical: https://www.zingnex.cn/forum/thread/qwen-sft
- Markdown 来源: floors_fallback

---

## Qwen Small Model Reasoning Ability Distillation Practice: Exploration of Combining SFT and On-Policy Distillation (Introduction)

This project aims to explore how to transfer the reasoning capabilities of large models to small Qwen models through the combination of Supervised Fine-Tuning (SFT) and on-policy distillation, in order to achieve efficient inference on edge devices. The core innovation lies in adopting an on-policy distillation mode of "learning by doing", allowing the student model to actively generate reasoning processes and optimize based on real-time feedback from the teacher model, breaking through the limitations of traditional methods. (Original author: kakopappa, Source: GitHub, Release date: 2026-06-13)

## Background: Dilemmas in Transferring Large Model Reasoning Capabilities and Limitations of Existing Methods

With the excellent performance of Large Language Models (LLMs) in complex reasoning tasks, how to transfer their capabilities to resource-constrained small models has become a focus of the industry. Traditional Supervised Fine-Tuning (SFT) can make small models imitate the output of large models, but it is difficult to acquire the internal reasoning chain; static distillation allows the student model to passively learn the "standard answers" of the teacher, which cannot be dynamically adjusted, limiting reasoning flexibility.

## Project Overview: Innovative Attempt of On-Policy Distillation

This project focuses on cultivating the reasoning capabilities of small models in the Qwen series, innovatively combining SFT and on-policy distillation. Unlike offline methods, on-policy distillation allows the student model to actively generate answers during training, with real-time evaluation and feedback from the teacher model—similar to the human process of "learning by doing", which is more suitable for the multi-path solution characteristics of reasoning tasks.

## Analysis of Core Technical Mechanisms

### Supervised Fine-Tuning (SFT) Phase
First, SFT is performed using high-quality reasoning datasets (including chain-of-thought annotations) to lay the foundation for the model's reasoning ability and help it understand reasonable reasoning steps and logical expressions.
### On-Policy Distillation Phase
1. Sampling and Generation: The student model generates multiple candidate answers for the problem;
2. Policy Evaluation: The teacher model evaluates the quality of candidate answers and provides reward signals;
3. Policy Optimization: The student model adjusts parameters based on rewards to optimize towards higher reward directions.
### Model Architecture and Training Strategy
The Qwen series is selected (considering Chinese-English balance and open licensing), and a curriculum learning strategy (from simple to complex tasks) is adopted to ensure training stability.

## Experimental Design and Evaluation Dimensions

The project evaluation covers four major dimensions:
- **Reasoning Accuracy**: Accuracy on mathematical reasoning benchmarks such as GSM8K and MATH;
- **Generation Quality**: Coherence and interpretability of the reasoning process;
- **Computational Efficiency**: Inference speed and memory usage (adapted for edge devices);
- **Generalization Ability**: Performance on reasoning tasks outside the training data to verify general reasoning capabilities.

## Practical Significance and Application Prospects

This work provides a feasible path for edge-side inference, solving the latency, privacy, and cost issues of cloud deployment. Specific application scenarios include:
- Smart assistants on mobile devices (no network required);
- Educational tutoring (real-time math problem solving and idea explanation);
- Lightweight code assistance in development environments;
- Edge real-time image reasoning and defect detection for industrial quality inspection.

## Technical Limitations and Future Directions

**Limitations**:
1. Training Stability: The online loop is sensitive to hyperparameters, easily diverging or converging to suboptimal solutions;
2. Dependence on Teacher Model: Distillation effect is affected by the quality of the teacher model;
3. Computational Overhead: More resource-intensive than pure SFT.
**Future Directions**: Introduce multi-teacher integration, efficient sampling strategies, and combine RLHF to optimize model behavior.

## Conclusion

The Qwen small model reasoning ability distillation experiment represents an important exploration direction for edge-side large model applications. Through the combination of SFT and on-policy distillation, it demonstrates a feasible path to cultivate the reasoning ability of small models under limited resources. As the demand for edge-side AI grows, such model compression and capability transfer research will become more important. We look forward to the continuous iteration of the project to provide practical experience and open-source resources for the community.