Empirical Study of Open-Source Lightweight Reasoning Models on Reasoning Tasks: Capabilities and Limitations

This article conducts an empirical study on open-source lightweight reasoning models, analyzing their performance characteristics on reasoning tasks, exploring the relationship between model size and reasoning ability, evaluating their practical application value, and pointing out current limitations and improvement directions. This study is of great significance to the process of AI democratization.

From late 2024 to early 2025, reasoning models represented by OpenAI's o1 and o3 series triggered an AI paradigm shift, improving the effectiveness of multi-step reasoning tasks by generating internal reasoning chains. However, these top models are mostly closed-source or high-cost. Whether the open-source community can reproduce this capability and how lightweight open-source models perform have become key issues for AI democratization.

The open-source community endows models with reasoning capabilities through multiple strategies:

1. Supervised Fine-Tuning (SFT)：Fine-tune base models with high-quality reasoning data to teach structured reasoning processes;
2. Reinforcement Learning：For example, GRPO (Group Relative Policy Optimization) guides effective reasoning strategies;
3. Inference-Time Computational Expansion：Increase computational budget during inference and improve performance through test-time training.

The experiment evaluates model performance from four dimensions:

• Logical Reasoning: Test the ability to follow formal logic rules (e.g., logic puzzles, syllogisms);
• Mathematical Reasoning: Cover basic arithmetic to medium-difficulty problems, requiring understanding of structure and strategies;
• Common Sense Reasoning: Use world knowledge for reasonable inferences;
• Multi-Step Reasoning: Evaluate the ability to maintain reasoning chains and avoid intermediate errors.

Experimental observations include:

1. Scale Effect: Among models in the 7B-14B parameter range, size is positively correlated with reasoning ability; models with <7B parameters struggle with complex tasks;
2. Reasoning Chain Quality: Some models have clear and coherent reasoning chains, while others have issues like jumps, circular arguments, hallucinatory reasoning, and premature termination;
3. Task Sensitivity: Models show large performance differences across different reasoning tasks, possibly related to the distribution of training data;
4. Prompt Sensitivity: High sensitivity to prompt engineering, and robustness needs improvement.

Technical Challenges:

• Coupling of reasoning and knowledge: Limited knowledge capacity of lightweight models restricts reasoning;
• Long-range dependency issue: Unstable attention when processing long sequences, prone to forgetting or contradictions;
• Weak self-correction ability: Difficult to detect and correct reasoning errors.

Practical Value:

• Edge deployment: Can run on consumer-grade hardware, suitable for privacy/network-constrained scenarios;
• Domain-specific fine-tuning: Can achieve acceptable performance in vertical domains;
• Reasoning teaching: Transparency is conducive to studying reasoning mechanisms;
• Cost-sensitive scenarios: Significant advantage in low-cost operation.

Future improvement directions include:

1. Data Quality Improvement: Synthetic data generation, expert-annotated dataset construction;
2. Architecture Optimization: Improve attention mechanisms, explicit reasoning state management, etc.;
3. Distillation and Transfer: Transfer capabilities from large closed-source models to lightweight models;
4. Multi-Model Collaboration: Different models take charge of different stages or aspects of reasoning.

Although open-source lightweight reasoning models have gaps compared to top closed-source models, they have unique advantages in accessibility, customizability, and cost-effectiveness. With technological progress, they will play an important role in AI democratization. Developers and researchers need to understand their capabilities and limitations and choose appropriate technical solutions.