Quality-Utility Paradox: Why High-Reward Data Harms Small Models' Mathematical Reasoning Ability

A paper accepted at ICML 2026 reveals a counterintuitive finding: high-reward data refined by strong models (Oracles) actually harms small models' mathematical reasoning ability more than data generated and filtered by the small models themselves. This phenomenon is called the 'Quality-Utility Paradox', and its core cause is that Oracle refinement leads to a drift in the small model's native reasoning distribution. The study proposes a style-aligned refinement method to address this issue.

Knowledge distillation is a common technique to enhance the capabilities of small language models (SLMs). In mathematical reasoning tasks, the mainstream approach is to use Oracles to generate high-quality reasoning trajectories for training student models. The core assumption is: the higher the reward model score of a trajectory, the better its quality and the better the distillation effect. This study challenges this assumption.

Experiments validate the 'Quality-Utility Paradox': the training effect of high-reward data refined by Oracles is consistently worse than that of data generated by small models themselves plus rejection sampling. This phenomenon exists across Qwen2.5, LLaMA-3, and DeepSeek series models, indicating it is a universal phenomenon rather than an exception.

Oracle refinement has dual effects: logical repair (correcting errors, positive) and distribution drift (changing reasoning style, deviating from the small model's native distribution, negative). Small models face a trade-off during learning: the benefit of logical repair vs. the cost of distribution adaptation. When the drift is large enough, the adaptation cost exceeds the benefit, leading to performance degradation.

Core idea: Logical correctness and reasoning style can be separated. Implementation steps: 1. Preserve the small model's native trajectory; 2. Use an Oracle or validator to locate errors; 3. Modify only the error steps while keeping other steps in their native expression; 4. Style consistency check. Effect: Reduces adaptation cost, preserves logical benefits, and outperforms baselines.

Experimental setup: Model families include Qwen2.5, LLaMA-3, DeepSeek; Data comparisons are Oracle refinement, self-generated + rejection sampling, style-aligned refinement; Evaluation metric is mathematical reasoning accuracy. Key findings: The paradox exists, drift quantification is significant, and the style-aligned method has the best performance.

Theoretical implications: Data quality needs to be redefined (perceived quality + learner compatibility), and a joint optimization framework should be adopted: Total utility = benefit of logical correctness - cost of distribution adaptation. Practical recommendations: 1. Use Oracle refinement cautiously; 2. Pay attention to the distribution matching between data and student models; 3. Try style-aligned refinement; 4. Take the final model performance as the gold standard for data quality.

Limitations: Only validated on mathematical reasoning tasks, style quantification is heuristic, and supervision is required. Future directions: Validate on other tasks (e.g., code generation), precise style quantification, and develop automated style-aligned methods.