Panoramic Analysis of Large Language Model Reasoning Capabilities: The Evolution from Chain-of-Thought to Reinforcement Learning

This article systematically reviews the development of large language model (LLM) reasoning technologies, from basic chain-of-thought prompting to the latest process reward model training. It covers key methods such as Self-Consistency, Tree-of-Thoughts, and Program-of-Thought, and compares the performance differences of various technical routes in tasks like mathematical reasoning and commonsense question answering based on comprehensive data from over 50 studies, providing a panoramic perspective for researchers and practitioners.

Although LLMs perform well in NLP benchmark tests, complex reasoning faces two core challenges: first, the hallucination phenomenon, where factual errors are easily generated and amplified in multi-step logical deduction; second, prompt sensitivity, where minor prompt changes can lead to 20%-40% accuracy fluctuations, affecting the stability of practical applications.

Chain-of-Thought (CoT) prompting is a milestone in improving LLM reasoning:

• Few-Shot CoT: Provides examples with reasoning processes, increasing GSM8K accuracy from 17.9% to 56.4% and MATH competition dataset accuracy from 5.2% to 18.7%;
• Zero-Shot CoT: Triggers reasoning through instructions like "Let's think step by step", achieving 40.7% accuracy on GSM8K, proving the inherent reasoning potential of LLMs.

A single reasoning path is prone to local optima; multi-path methods solve this problem:

• Self-Consistency: Samples multiple reasoning paths and uses majority voting, increasing GSM8K accuracy to 74.4% and MATH to 33.9%;
• Tree-of-Thoughts: Models the reasoning process with tree search, achieving 79.3% on GSM8K and 82.0% on StrategyQA commonsense reasoning, but with increased computational overhead.

LLMs are inaccurate in arithmetic calculations; tool enhancement methods are effective:

• Program-of-Thought: Generates executable code (e.g., Python) and obtains precise results using an external interpreter, achieving 57.0% accuracy on the MATH dataset;
• Extension directions: Calling calculators, Python interpreters, external knowledge bases, APIs, etc.

The latest progress in reasoning training adopts process reward models:

• Fine-grained evaluation of each reasoning step, with step-level reinforcement learning supervision improving complex reasoning performance;
• o1-style models achieve 92.4% on GSM8K, 83.3% on MATH, and 88.5% on StrategyQA, approaching human expert levels;
• Advantages: Fine-grained training signals, identifying error locations, guiding effective strategies, and reducing reliance on manual annotations.

Key Findings: Chain-of-thought is a foundational technology; multi-path reasoning improves performance but increases computational costs; tool enhancement solves arithmetic accuracy issues; process reward models represent the current state-of-the-art; Future Directions: Formal verification (integrating Lean/Coq), memory-enhanced architectures, causal reasoning, multi-modal reasoning, knowledge distillation, etc.

LLM reasoning technologies have evolved from simple prompt engineering to complex training methods, with each step expanding the boundaries of AI reasoning. Practitioners need to understand the applicable scenarios and trade-offs of each technology, while researchers can focus on cutting-edge areas such as formal verification and causal reasoning.