Qwen Small Model Reasoning Ability Distillation Practice: Exploration of Combining SFT and On-Policy Distillation

This project aims to explore how to transfer the reasoning capabilities of large models to small Qwen models through the combination of Supervised Fine-Tuning (SFT) and on-policy distillation, in order to achieve efficient inference on edge devices. The core innovation lies in adopting an on-policy distillation mode of "learning by doing", allowing the student model to actively generate reasoning processes and optimize based on real-time feedback from the teacher model, breaking through the limitations of traditional methods. (Original author: kakopappa, Source: GitHub, Release date: 2026-06-13)

With the excellent performance of Large Language Models (LLMs) in complex reasoning tasks, how to transfer their capabilities to resource-constrained small models has become a focus of the industry. Traditional Supervised Fine-Tuning (SFT) can make small models imitate the output of large models, but it is difficult to acquire the internal reasoning chain; static distillation allows the student model to passively learn the "standard answers" of the teacher, which cannot be dynamically adjusted, limiting reasoning flexibility.

This project focuses on cultivating the reasoning capabilities of small models in the Qwen series, innovatively combining SFT and on-policy distillation. Unlike offline methods, on-policy distillation allows the student model to actively generate answers during training, with real-time evaluation and feedback from the teacher model—similar to the human process of "learning by doing", which is more suitable for the multi-path solution characteristics of reasoning tasks.

Supervised Fine-Tuning (SFT) Phase

First, SFT is performed using high-quality reasoning datasets (including chain-of-thought annotations) to lay the foundation for the model's reasoning ability and help it understand reasonable reasoning steps and logical expressions.

On-Policy Distillation Phase

1. Sampling and Generation: The student model generates multiple candidate answers for the problem;
2. Policy Evaluation: The teacher model evaluates the quality of candidate answers and provides reward signals;
3. Policy Optimization: The student model adjusts parameters based on rewards to optimize towards higher reward directions.

Model Architecture and Training Strategy

The Qwen series is selected (considering Chinese-English balance and open licensing), and a curriculum learning strategy (from simple to complex tasks) is adopted to ensure training stability.

The project evaluation covers four major dimensions:

• Reasoning Accuracy: Accuracy on mathematical reasoning benchmarks such as GSM8K and MATH;
• Generation Quality: Coherence and interpretability of the reasoning process;
• Computational Efficiency: Inference speed and memory usage (adapted for edge devices);
• Generalization Ability: Performance on reasoning tasks outside the training data to verify general reasoning capabilities.

This work provides a feasible path for edge-side inference, solving the latency, privacy, and cost issues of cloud deployment. Specific application scenarios include:

• Smart assistants on mobile devices (no network required);
• Educational tutoring (real-time math problem solving and idea explanation);
• Lightweight code assistance in development environments;
• Edge real-time image reasoning and defect detection for industrial quality inspection.

Limitations:

1. Training Stability: The online loop is sensitive to hyperparameters, easily diverging or converging to suboptimal solutions;
2. Dependence on Teacher Model: Distillation effect is affected by the quality of the teacher model;
3. Computational Overhead: More resource-intensive than pure SFT. Future Directions: Introduce multi-teacher integration, efficient sampling strategies, and combine RLHF to optimize model behavior.

The Qwen small model reasoning ability distillation experiment represents an important exploration direction for edge-side large model applications. Through the combination of SFT and on-policy distillation, it demonstrates a feasible path to cultivate the reasoning ability of small models under limited resources. As the demand for edge-side AI grows, such model compression and capability transfer research will become more important. We look forward to the continuous iteration of the project to provide practical experience and open-source resources for the community.