New Findings in Reasoning Data Quality Assessment: Model Scale Determines Optimal Data Filtering Strategy

Researchers have found that the quality prediction metrics for reasoning training data are scale-dependent—small models require precisely aligned data, while large models benefit from detailed reasoning chains with high redundancy. This finding provides a practical framework for data filtering before training reasoning models, helping reduce trial-and-error costs and improve R&D efficiency.

Currently, the improvement of reasoning capabilities in large language models mainly relies on Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL), but the success of these methods is highly dependent on the quality of training data. Verifying the effectiveness of reasoning datasets requires expensive trial-and-error experiments (training first, then validation), and the high cost restricts R&D efficiency. Traditional heuristic rules (such as answer length, format standardization) lack theoretical support and cannot accurately predict downstream performance. The industry urgently needs an intrinsic measurement system that can reliably predict data utility before training.

The study reveals that the metrics for predicting reasoning data utility are significantly scale-dependent. Experiments fine-tuned models with 8B and 11B parameters on variants of the Polish reasoning dataset and found that models of different scales have inherently different sensitivities to data features: small models (e.g., 8B) require highly precise, uniformly formatted, and logically rigorous aligned data, and redundant information will interfere with learning; large models (e.g., 11B) can benefit from detailed multi-step reasoning chains with high redundancy and extract deep reasoning patterns.

Experimental framework: Construct a Polish reasoning dataset and multiple semantic variants (differing in alignment, redundancy, reasoning chain length, etc.); propose quantitative intrinsic data metrics (alignment metric, redundancy metric, complexity metric, diversity metric); record performance by training models, establish a statistical correlation model between intrinsic metrics and downstream quality, and verify that the metrics can reliably predict performance but their weights change with model scale.

Practical guidance: Small model teams should prioritize data that is format-standardized, logically precise, and highly aligned with the target task, avoiding lengthy or redundant reasoning examples; large model teams can relax conciseness requirements and collect data with rich reasoning details (such as detailed thought chains, multi-angle analysis); the framework can evaluate dataset value before training, avoid expensive trial-and-error, accelerate iterative development, and lower R&D thresholds.

Limitations: The experiments were mainly conducted on Polish datasets, and whether the conclusions apply to other languages needs to be verified; only 8B and 11B models were tested, and the behavior of larger-scale models (e.g., 70B+) may be different; only the SFT scenario was focused on, and whether the data filtering strategy in the RL training phase follows the same rules remains to be explored. Future directions: Expand to multilingual environments, larger-scale models, and RL training scenarios.

Core conclusion: There is no one-size-fits-all data quality standard; the optimal data filtering strategy depends on the target model scale. Small models need precisely aligned data, while large models benefit from detailed reasoning trajectories. This scale-aware framework is expected to help researchers build high-quality reasoning training datasets more efficiently and promote the development of reasoning model technology.