OPSD: Post-RL Compression Technology for Reasoning Models

OPSD (Online Policy Self-Distillation) is a post-RL compression technology for reasoning models, designed to address the issues of large parameter size and high inference cost of reasoning models trained via reinforcement learning. This technology adds a compression stage after RL training to distill the knowledge of large models into smaller ones, achieving both performance preservation and improved inference efficiency. The project is maintained by jaeh8nkim, with source code available on GitHub (https://github.com/jaeh8nkim/compressor), and was released in May 2026.

In recent years, RL-based reasoning models (such as DeepSeek-R1, OpenAI o1/o3 series) have shown outstanding performance in tasks like mathematics and code, but their parameter sizes are enormous (billions to hundreds of billions), leading to high inference costs and deployment thresholds. Traditional knowledge distillation struggles to fully preserve the complex reasoning patterns acquired via RL. The core question is: How to reduce model size and inference cost while maintaining reasoning capabilities?

OPSD adopts a two-stage training paradigm:

1. RLVR Training: Use Reinforcement Learning with Validation Reward (RLVR) to train a powerful teacher model, learning complex reasoning strategies;
2. OPSD Compression: The core innovative stage, which compacts the knowledge of the teacher model into a small model via online policy self-distillation, preserving key RL capabilities.

Implementation Details:

• Architecture Components: verl framework (supports distributed training), workspace experiment configuration;
• Environment Requirements: 4/8x H100/H200 GPU, Linux + CUDA12.2 + PyTorch2.9.1;
• Installation Process: Clone the repository → Create a conda environment → Install verl and dependencies.

Core Advantages:

1. Performance Preservation: Close to the teacher model on benchmarks like GSM8K and MATH, with multi-step reasoning better than supervised fine-tuning;
2. Efficiency Improvement: Parameter size reduced by 50%-90%, inference latency and memory usage decreased;
3. Deployment-Friendly: Supports vLLM/TensorRT-LLM, seamlessly integrates with existing services.

Application Scenarios:

• Edge Devices: Local inference on mobile phones/IoT;
• Production Environment: Cost optimization for API services, improved high-concurrency throughput;
• Prototype Development: Quickly obtain deployable small models from large model RL training.

Experimental Evaluation Dimensions:

• Benchmark Tests: Mathematics (GSM8K/MATH), Code (HumanEval), Logic (BBH);
• Efficiency Metrics: Inference latency, memory usage, throughput;
• Compression Ratio Experiments: Performance curves under different ratios.

Potential Challenges:

• Training Cost: Requires RLVR + compression stages, high demand for multi-GPU computing resources;
• Generality: Mainly optimized for reasoning tasks, effectiveness in creative writing etc. remains to be verified;
• Complexity: Relies on a modified VERL framework, distributed configuration is complex.

Industry Insights: OPSD represents a new direction for model efficiency optimization—balancing deployment efficiency and capability, aligning with trends:

1. Revival of distillation technology;
2. Inference efficiency is as important as training;
3. Hierarchical deployment (large model training, small model serving).

Future Outlook:

• Technical Improvements: More aggressive compression, combining quantization and pruning, adaptive strategies;
• Application Expansion: Multimodal, long context, real-time inference;
• Ecosystem Construction: Pre-compressed models, one-click tools, evaluation benchmarks.