Build Reasoning Model: Open Source Practice for Reproducing DeepSeek-R1's Reasoning Capabilities Based on the GRPO Algorithm

This project aims to democratize the GRPO training method of DeepSeek-R1 through algorithm optimization and engineering techniques, enabling ordinary developers to reproduce reasoning model training on consumer-grade hardware. It primarily adopts the Unsloth optimization framework, 4-bit quantization, and LoRA fine-tuning technology to compress the training scale to run on the free version of Google Colab (T4 GPU), and enhance reasoning capabilities based on pre-trained models.

In early 2025, the open-source DeepSeek-R1 reasoning model's performance was close to OpenAI o1, but its training required thousands of H100 GPUs, which was out of reach for ordinary developers. The build-reasoning-model project emerged as a result, with the mission to prove that GRPO training can be democratized through optimization. It uses the Unsloth framework combined with quantization and LoRA technology to compress the training task to a scale that can run on consumer-grade hardware, with the strategy of 'small model + efficient algorithm'.

GRPO is a reinforcement learning algorithm proposed by DeepSeek. Compared to traditional PPO, it has three major innovations: 1. Intra-group relative reward: Generate multiple answers (a group) for the same problem, distribute rewards based on relative quality, reducing reliance on a separate reward model; 2. Reasoning process supervision: The reward function encourages complete reasoning chains, enhancing thinking ability; 3. Computational efficiency optimization: Reduce overhead through sampling strategies and gradient estimation, suitable for large-scale model training.

Model Selection: The high-performance solution uses Llama-3.1-8B-Instruct (4-bit quantization + LoRA), requiring 14GB of VRAM; the inclusive solution uses Qwen2.5-Math-7B-Instruct (no authorization required, runs on T4, significant improvement in math tasks), and there is also a 1.5B fallback model. Unsloth Framework: Through kernel fusion, quantization-aware training, and gradient checkpoint optimization, it speeds up training by 2-5 times and reduces VRAM usage by 30-70%. LoRA Fine-tuning: Inject low-rank matrices into key layers to save VRAM, maintain stability, and enable flexible deployment. Memory Configuration: Adjust parameters (such as batch size=1, gradient accumulation=4, etc.) for Colab T4 to balance VRAM usage and performance.

The project uses GSM8K (8,000 elementary school math problems) as the training dataset. Its characteristics: verifiable answers, requiring multi-step reasoning, moderate difficulty. It is a standard benchmark for evaluating reasoning capabilities and suitable for GRPO training (emphasizing reasoning process rather than guessing).

The training process includes: 1. Environment preparation: Install Unsloth and vLLM dependencies; 2. Model loading: Use Unsloth to load the 4-bit quantized model and configure LoRA; 3. Data preparation: Format the GSM8K dataset; 4. GRPO configuration: Set hyperparameters (learning rate, batch size, etc.); 5. Reward function design: Reward based on answer correctness; 6. Training execution: Monitor loss and rewards; 7. Model export: Merge LoRA weights or save the adapter.

The project's value includes: Educational value (a practical platform to understand GRPO); Research value (low-cost verification of new strategies); Application value (specific scenarios such as math tutoring); Community value (open source promotes knowledge sharing and lowers the threshold for cutting-edge research).

Limitations: Small model size (7B/1.5B vs DeepSeek-R1's 671B); limited data (only GSM8K math reasoning); hardware still requires more than 15GB of VRAM. Future Directions: Support more datasets (code, scientific Q&A); explore model merging/distillation; more efficient quantization schemes; build reasoning evaluation benchmarks.