SSD: Simple Self-Distillation Significantly Improves Code Generation Capability

Simple Self-Distillation (SSD) improves code generation capability through sampling with specific temperature configurations plus standard supervised fine-tuning training, without needing validators, teacher models, or reinforcement learning. On LiveCodeBench, SSD increases the pass@1 of Qwen3-30B-Instruct from 42.4% to 55.3%. The method is concise and general, applicable to various models and scales.

Large language models have demonstrated strong code generation capabilities, but traditional post-training methods rely on external resources: reinforcement learning requires complex reward functions, distillation needs stronger teacher models, and validators demand code execution environments. These dependencies increase complexity and limit scalability, raising a core question—can models improve solely based on their own outputs?

The core process of SSD consists of only two steps: 1. Sample solutions from the model itself using specific temperature and truncation configurations; 2. Perform standard supervised fine-tuning using these samples. Its assumption is that the model already knows the correct answer and needs more reliable outputs. High-temperature sampling explores diverse solutions, and screening followed by fine-tuning consolidates effective patterns. SSD is concise and general, can be implemented on standard infrastructure, and is applicable to models of all scales/types.

SSD has significant effects: On LiveCodeBench v6, the pass@1 of Qwen3-30B-Instruct increases by more than 12 percentage points (from 42.4% to 55.3%), with gains concentrated on complex multi-step reasoning problems. It has strong generalization: applicable to Qwen/Llama series, 4B-30B scales, instruction/reasoning models, and touches on the fundamental principles of code generation.

SSD resolves the conflict between accuracy and exploration in LLM decoding: High-temperature sampling explores diverse solutions; after screening correct samples for fine-tuning, it reshapes token distribution and achieves context-dependent adjustments (concentrate on precise areas, maintain diversity in areas needing exploration). No external validator needed: Use test case execution results to screen correct samples; the validation process is fast and reliable, and training is standard supervised learning, reducing costs.

Compared with reinforcement learning: Simpler and more stable, avoiding reward function design and training instability; compared with distillation: More autonomous and general, no need for external teacher models; compared with validator methods: More efficient and flexible, only validating during training, no extra steps in inference.

Application recommendations: Choose sampling temperature between 0.8-1.2, use top-p/top-k truncation; generate dozens to hundreds of samples per problem; use small learning rate and regularization for fine-tuning. Limitations: Relies on test case screening, limited for tasks without clear test standards; mainly improves pass@1. Future directions: Iterative/multi-round self-distillation, expansion to tasks like mathematical reasoning, combination with other post-training methods.

Enlightenment: Simple methods may be the most effective; AI can self-improve (learn from its own outputs); need to focus on fundamental principles. Conclusion: SSD achieves significant results with simple technology, challenges inherent cognition, provides developers with ready-to-use tools, and more innovative self-distillation methods will emerge in the future.