LLM Math Hallucination Evaluator: Unveiling Algebraic Vulnerabilities of Large Language Models via Perturbation Testing

This article introduces the llm-math-hallucination-evaluator project, which systematically detects the 'surface dependency' hallucinations of large language models in mathematical reasoning through a perturbation testing framework. The study found that the overall accuracy rate of models is only 57.6%, showing significant systematic vulnerability, and verified that intervention strategies such as symbolic scaffolding can effectively improve model reliability.

Traditional hallucination research mostly focuses on factual errors, while mathematical hallucinations are more subtle: models' answers fluctuate under different expressions of equivalent mathematical problems, reflecting their reliance on surface forms rather than deep logic. The core insight of this project is: a model that truly understands mathematics should give consistent answers to equivalent problems; otherwise, it has 'surface dependency' hallucinations.

The project's technical architecture consists of three parts:

1. Perturbation Engine: Generates 10 expression variants (6 semantic-preserving, 4 adversarial traps) to test model consistency;
2. Hallucination Classification System: Classifies errors into 6 categories, such as external variable invention (most frequently triggered by identity multiplication traps), domain transformation, etc.;
3. Evaluation Metrics: Expression Consistency Score (ECS), accuracy rate, robustness score (weighted by consistency and accuracy).

Results from large-scale experiments (900 queries):

• Overall accuracy rate is only 57.6%;
• Adversarial traps are significantly destructive (identity multiplication triggered 52 external variable invention errors);
• Symbolic Scaffolding Strategy: Completely eliminated hallucinations in the DeepSeek-Chat model, with a robustness score of 1.0.

The project is developed using Python, relying on the SymPy library for symbolic math parsing and canonical normalization (serving as the 'gold standard' for answer correctness); it integrates models from multiple LLM providers via the OpenRouter API, simplifying the cross-model comparison process.

The research has important value in multiple fields:

• Education: Helps screen reliable AI learning tools to avoid misleading students;
• Code Generation: Guides developers in designing prompts and verification mechanisms;
• Scientific Research: Provides a reliability assessment tool for AI-assisted data analysis.

This project reveals that LLM mathematical reasoning has systematic vulnerabilities, but intervention strategies like symbolic scaffolding are effective. Future directions include: refining hallucination classification, exploring more intervention methods, and expanding evaluation to mathematical fields such as geometry/probability. This framework helps practitioners evaluate model capability boundaries and make informed technical choices.