Hidden Bottleneck in Arithmetic Reasoning of Small Language Models: Format Compliance Rather Than Reasoning Ability

Latest research reveals that the core reason for the poor performance of small language models (SLMs) in arithmetic reasoning tasks is not the lack of reasoning ability, but the difficulty in meeting strict output format requirements. Traditional evaluation methods may systematically underestimate the true capabilities of small models due to format constraints, and this finding has important implications for model evaluation, application optimization, and training directions.

Against the backdrop of the rapid development of LLMs, SLMs have attracted attention due to their low deployment cost and fast inference speed, but they have long had the problem of poor arithmetic reasoning performance. The traditional view holds that the lack of model capacity leads to reasoning ability defects, while the latest research proposes a new perspective: failures may stem from output format compliance requirements masking true capabilities.

The research team selected open-source SLMs with 1B-7B parameters and conducted two groups of experiments on the GSM8K and SVAMP datasets:

Strict Format Group: Required to output reasoning processes and answer markers according to a preset template; Loose Format Group: Only required correct answers with no format restrictions.

The results show: the average accuracy of the strict format group was only 23%, while the loose group jumped to 61%; more than 70% of the errors in the strict group were format parsing failures, not calculation errors.

Small models have limited parameters; when handling both reasoning and format control, they need to allocate limited cognitive resources, and format control consumes reasoning resources; in addition, the distribution of structured outputs in training data is scarce, leading to weak format capabilities.

The implications include:

1. Evaluation level: Treat format compliance as an independent dimension and adopt flexible evaluation strategies;
2. Application level: Separate reasoning and format conversion (e.g., post-processing modules);
3. Training level: Increase structured output samples or strengthen format control techniques.

Limitations: The experiments are focused on arithmetic reasoning and need to be verified on other tasks; the impact of input formats was not explored. Future directions: Quantify the resource consumption of format control and explore small model optimization techniques (e.g., knowledge distillation).

Evaluation methodologies may introduce biases, and the arithmetic problems of small models may be expression problems. Developers may already have stronger small models and need to find the key to unleashing their potential.