Practical Guide to LLM Distillation and Fine-Tuning: A Complete Technical Roadmap from SFT to GRPO

This article introduces an open-source project for LLM optimization covering supervised fine-tuning (SFT), GRPO reinforcement learning, and multimodal model fine-tuning. It provides optimized scripts and a complete evaluation toolchain for Qwen series models, aiming to address the core challenge of balancing LLM performance and efficiency.

With the exponential growth of Large Language Model (LLM) scale, how to reduce inference costs while maintaining performance has become a core challenge in the AI engineering field. Model distillation and fine-tuning, as two key technical paths, provide practical solutions to this problem. This article will deeply introduce a complete technical practice project covering from unimodal to multimodal models.

The project offers three core capabilities: 1. Supervised Fine-Tuning (SFT): Adapt pre-trained models to specific domains using high-quality annotated data; 2. GRPO Reinforcement Learning: Adopt the Group Relative Policy Optimization algorithm, which does not require a value network, improving memory efficiency and training stability; 3. Multimodal Fine-Tuning: Support joint training of vision-language models. GRPO has significant advantages over traditional PPO, including simplifying the process by eliminating the value network, reducing GPU memory usage, and more stable training. Additionally, it has been deeply optimized for Qwen series models with techniques like gradient accumulation and dynamic learning rate scheduling.

For the Qwen series models open-sourced by Alibaba Cloud, the project has implemented several targeted optimizations: Attention mechanism adaptation (adjusting SwiGLU activation function and RoPE-related hyperparameters); Chinese tokenization optimization (optimizing preprocessing based on BPE tokenizer features); Long context support (providing 32K/128K version fine-tuning scripts, including position encoding extrapolation and dynamic NTK scaling techniques).

The project supports fine-tuning of vision-language models like Qwen-VL, with application scenarios including image-text understanding, visual question answering, and document analysis. Technically, it uses the LoRA efficient fine-tuning method, which can achieve significant performance improvements by training only a small number of adapter parameters while retaining the general capabilities of the base model.

The project provides a multi-dimensional evaluation toolchain: Automatic metric evaluation (BLEU, ROUGE, Perplexity, C-Eval, CMMLU, etc.); Manual evaluation framework (standardized interface and scoring criteria, supporting A/B comparison tests); Inference performance testing (measuring inference latency and throughput on different hardware).

Based on project practice experience, the following recommendations are summarized: 1. Prioritize data quality (invest at least 60% of effort in cleaning and annotation); 2. Progressive training (first build basic capabilities with SFT, then optimize via GRPO); 3. Hyperparameter sensitivity (systematic grid search is recommended); 4. Continuous evaluation (regularly save checkpoints and evaluate).

This project not only provides runnable code but also demonstrates the best practice paradigm in the LLM optimization field. The complete technical chain from distillation to fine-tuning, unimodal to multimodal, and training to evaluation provides valuable references for the community. Open-source projects will become an important force driving the democratization of large model technology.