Single-GPU Training for Reasoning Models: mini-grpo Implements Core Algorithm of DeepSeek-R1

The mini-grpo project implements the GRPO algorithm with minimal code, allowing researchers and developers to reproduce the reinforcement learning training process of DeepSeek-R1 on a single GPU. This project lowers the resource barrier for cutting-edge LLM training technologies, facilitates algorithm understanding and modification, and helps the community explore reasoning model optimization.

Evolution from PPO to GRPO

Traditional LLM reinforcement learning fine-tuning relies on the PPO algorithm, but the critic network itself is a large model, leading to huge memory overhead and high training costs. The core insight of the GRPO algorithm is to estimate advantages using intra-group relative performance, eliminating the need for an additional critic network and significantly reducing computational resource requirements.

Design Philosophy

mini-grpo follows the "minimal, hackable" philosophy, with a streamlined codebase and clear, readable core logic. Unlike complex frameworks, it exposes the essence of GRPO (data loading, reward calculation, policy update, etc.), making it easy for learners to understand the principles and for researchers to quickly experiment with algorithm variants.

Memory Optimization and Training Flow

Key technologies for single-GPU training include: gradient accumulation, activation checkpointing (computation in exchange for memory), and 8-bit optimizer state compression. The training flow is simplified as: generate candidate outputs → reward scoring → calculate intra-group relative advantages → update policy. Without the need for a critic network, memory requirements are halved.

Experiments and Scenarios

The project provides a training example using the GSM8K math dataset, suitable for improving math, code, and logical reasoning capabilities. It can train models with billions of parameters on consumer GPUs (e.g., RTX4090), and the base model needs to have basic instruction-following and generation capabilities to benefit from this.

Experimental Recommendations

1. Validate the correctness of the process with small-scale experiments first; 2. Gradually expand data size and training steps; 3. Fine-tune on task-specific data. The documentation provides strategies for selecting hyperparameters such as learning rate, batch size, and number of generated samples.

Project Significance

mini-grpo captures the core mechanism of GRPO and helps understand the details of DeepSeek-R1 (the differences lie in data scale and distributed training). It democratizes cutting-edge LLM training technologies, allowing more developers to access the application of reinforcement learning in LLMs and promoting knowledge dissemination.

Limitations and Future

Production features such as multi-GPU distribution and advanced monitoring are omitted. In the future, it can support more RL variants, integrate the vLLM inference engine, enrich reward functions, and optimize task-specific configurations, relying on community contributions.