Combining MCTS with Process Preference Model: Building a New Paradigm for Mathematical Reasoning in Large Language Models

This project innovatively combines Monte Carlo Tree Search (MCTS) with a process preference model, aiming to address core challenges faced by large language models in mathematical reasoning, such as broken reasoning chains, lack of verification mechanisms, and search space explosion. It significantly improves the accuracy of solving complex mathematical problems and opens up a new path for LLM mathematical reasoning.

Mathematical reasoning is an important standard to test the intelligence level of AI, but current mainstream LLMs face three major challenges in this field:

1. Broken Reasoning Chains: When solving complex multi-step problems, intermediate errors are difficult to self-correct;
2. Lack of Verification Mechanism: Autoregressive generation lacks validation of intermediate step effectiveness, easily leading to wrong paths;
3. Search Space Explosion: The mathematical solution space is huge, and greedy strategies struggle to find optimal solutions.

Monte Carlo Tree Search (MCTS)

The tree structure is designed as: root node (original problem) → internal nodes (intermediate steps) → edges (reasoning actions) → leaf nodes (complete path); iterative search through four stages: selection (UCB1 algorithm), expansion (LLM generates next step), simulation (fast rollout), and backpropagation (updates node value).

Process Preference Model

Focuses on intermediate step evaluation: step-level correctness judgment, contrastive learning to distinguish between good and bad steps, fine-grained feedback to prune wrong paths; training uses positive samples (correct intermediate steps), negative samples (wrong steps), and contrastive loss for optimization.

Synergistic Effect

MCTS provides search capabilities to explore the solution space, the process preference model provides high-quality evaluation to guide the search, and the search data further optimizes the model to form a closed loop.

Problem Analysis Phase

Semantic understanding to extract known conditions and goals → formal conversion to structured mathematical representation → difficulty assessment to dynamically adjust search parameters.

Reasoning Search Phase

Initialize root node → multiple rounds of MCTS iteration (selection/expansion/simulation/backpropagation) → LLM generates candidate steps → process preference model evaluates and filters → selects optimal path.

Result Verification Phase

Symbolic verification (computer algebra system) → numerical verification (reverse substitution) → logical consistency check.

Benchmark Tests

Evaluated on GSM8K (elementary school math), MATH (high school competition), and Olympiad-level (olympiad difficult problems) datasets.

Performance Improvement

• GSM8K: From approximately 70% to over 85%;
• MATH: From approximately 40% to around 60%;
• More significant improvement on complex multi-step problems.

Ablation Experiments

• Contribution of MCTS: Approximately 15% improvement over greedy decoding;
• Contribution of process preference model: Additional approximately 10% improvement when replacing result verification;
• Synergistic effect: Combined effect is better than using each alone.

Education Field

Intelligent tutoring tools: step-by-step explanation of problem-solving ideas, error diagnosis, adaptive practice.

Scientific Research Assistance

Formula derivation, proof exploration, model verification.

Technical Expansion

Multimodal reasoning (combining images), formal proof (combining with Lean/Coq), cross-domain applications (physics/chemistry, etc.).

This project, through the innovative combination of MCTS and process preference model, provides an interpretable and reliable technical path for LLM mathematical reasoning, significantly enhancing the ability to solve complex problems. This paradigm is not only applicable to the mathematical field but also provides valuable references for building general AI reasoning systems, and is expected to achieve greater breakthroughs in mathematics and more fields in the future.