Analysis of Mathematical Reasoning Capabilities of Large Language Models: Prompt Engineering Practice with Mistral-7B

This study conducts a systematic analysis of the multi-step mathematical reasoning capabilities of the open-source Mistral-7B model. By comparing various prompt strategies such as zero-shot prompting, few-shot prompting, Chain-of-Thought (CoT), zero-shot CoT, and self-consistency sampling, we explore their impact on the model's problem-solving performance. Key findings include: prompt strategies significantly affect model performance; Chain-of-Thought can effectively improve accuracy; few-shot prompting has an effectiveness threshold; self-consistency sampling can enhance result reliability. Additionally, common error patterns of the model are identified (arithmetic calculation errors, reasoning jumps, misinterpretation of problem statements, etc.). The research results provide practical guidance for the optimized use of open-source models in mathematical reasoning tasks.

Mathematical reasoning is an important benchmark for measuring the intelligence level of large language models, requiring rigorous logical deduction, precise symbol manipulation, and multi-step decomposition capabilities. However, mainstream models still have systematic flaws in deep reasoning of mathematical problems. As a small-parameter model of interest to the open-source community, Mistral-7B's performance is close to some large models, but there is a lack of systematic empirical research on its mathematical reasoning capabilities and the impact of prompt strategies. This project aims to fill this gap.

Model Selection: Mistral-7B is selected because its parameter scale (7B) balances computational efficiency and performance, it uses innovative architectures such as sliding window attention, and it is open-source and reproducible.

Dataset Construction: Covers multi-step mathematical problems in multiple fields such as algebra, geometry, probability and statistics, with moderate difficulty. Each problem is equipped with a standard answer and detailed steps to facilitate evaluation and analysis.

Prompt Strategies: Five strategies are compared:

• Zero-shot prompting: Directly present the problem to reflect native capabilities;
• Few-shot prompting: Provide examples of similar problems for guidance;
• Chain-of-Thought (CoT): Require displaying intermediate reasoning steps;
• Zero-shot CoT: Induce the reasoning process through trigger sentences;
• Self-consistency sampling: Take high-frequency answers from multiple samples to improve reliability.

Overall Performance: Mistral-7B's mathematical reasoning ability depends on prompt strategies, and the optimal configuration is significantly better than the baseline.

Comparison of Prompt Strategies:

• CoT effectively improves accuracy; explicit reasoning reduces error accumulation;
• The effect of few-shot prompting is not monotonically increasing; performance plateaus or declines after exceeding the threshold;
• Self-consistency sampling stably improves accuracy and is suitable for high-accuracy scenarios.

Error Patterns:

• Arithmetic calculation errors (large number/fraction operations);
• Reasoning step jumps (broken logical chain);
• Misinterpretation of problem statements (reasoning based on wrong assumptions);
• Symbol manipulation errors (algebraic transformation/equation solving errors).

Model Capability Boundaries: There is a gap between the model's native capabilities and its performance; appropriate prompts are needed to unlock potential, and the optimal usage should be explored during evaluation.

Value of Prompt Engineering: In resource-constrained scenarios, well-designed prompt strategies can effectively improve performance; prompt templates need to be optimized during deployment.

Competitiveness of Open-Source Models: Although Mistral-7B's parameter count is much smaller than closed-source large models, it can reach a practical level in specific tasks after optimization, making it suitable for cost and privacy-sensitive scenarios.

Limitations: The dataset does not cover all types and difficulty levels of mathematical problems; the exploration of the model's internal mechanisms is limited; results are affected by model versions and implementation details.

Future Directions: Combine tools (such as Python interpreters) to enhance computational accuracy; study the impact of multi-modal inputs (charts/formula images); explore the effect of fine-tuning; develop automatic prompt optimization methods.

This study deeply analyzes Mistral-7B's mathematical reasoning performance through systematic experiments, enhances understanding of the model's capabilities, and provides practical guidance for large language models to solve mathematical problems. As a core challenge of AI, mathematical reasoning still requires more exploration, and this study is an important step in this journey.