Introduction: SenseMath—A Benchmark Framework for Evaluating Mathematical Intuition of LLMs

SenseMath is an open-source benchmark tool focused on evaluating the numerical perception (mathematical intuition) capabilities of large language models (LLMs). It addresses the problem that traditional math tests only focus on computational ability while ignoring deep intuition. Through multi-dimensional design connecting cognitive science and AI, it helps reveal whether models truly understand mathematical concepts rather than relying on pattern matching.

Project Background and Motivation

Definition of Numerical Perception

Numerical perception is an innate cognitive ability of humans, including quantity intuition, numerical comparison, approximate estimation, and conservation of quantity. For LLMs, this means the ability to understand more vs. less, judge size without calculation, and reasonably estimate numerical ranges.

Limitations of Existing Evaluations

Traditional math benchmarks (e.g., GSM8K, MATH) focus on computation and problem-solving skills, ignoring numerical perception. This leads to models possibly scoring high on standard tests but making mistakes in simple quantity judgments, making it difficult to distinguish between reasoning and memory.

SenseMath Core Design

Evaluation Dimensions

1. Quantity Representation: Tests the model's accurate representation of different quantities, including small quantity recognition, large quantity estimation, and the association between numbers and concepts.
2. Numerical Comparison: Evaluates classic cognitive phenomena such as distance effect and size effect.
3. Quantity Operation: Tests the impact of addition/subtraction, conservation of quantity, and proportional reasoning ability.

Test Tasks

Includes tasks such as dot matrix comparison, numerical distance judgment, conservation of quantity, and approximate arithmetic, simulating human cognitive test scenarios.

Technical Implementation Details

Dataset Construction

Follows strict standards: single-dimensional evaluation, difficulty gradient, non-training corpus, and human comparison benchmark.

Evaluation Metrics

Uses multi-dimensional metrics such as correct answer ratio, error type consistency, confidence matching degree, and cross-task transfer ability.

Model Comparison

Supports standardized comparison of models with different architectures, parameter scales, and specialized/general training.

Research Findings and Insights

Current Status of LLMs

Most models perform well with 1-3 objects (consistent with human subitizing), but accuracy drops beyond the threshold; there is a large difference in how they handle Arabic numerals vs. dot matrices, relying on statistical patterns in training data rather than internal representation.

Model Design Insights

Pure text pre-training is insufficient; dedicated modules are needed; combine visual and symbolic training; design architectures by drawing on human cognitive laws.

Application Scenarios

Model Selection Guidance

Helps select models suitable for math tutoring, numerical data processing, and numerical simulation.

Model Improvement Directions

Add training data for weak points, design dedicated numerical modules, and integrate specialized computing engines.

Cognitive Science Research

Provides tools for human-AI comparison, simulation of model capability development, and internal activation analysis.

Limitations and Future Work

Existing Limitations

• Focuses on basic numerical perception; advanced mathematical intuition remains to be developed;
• Based on Western cognitive research, may not be applicable to all cultures;
• Lacks dynamic tracking of the model learning process.

Future Plans

• Expand complex concepts such as fractions and negative numbers;
• Develop adaptive tests;
• Establish multi-cultural datasets;
• Explore neuro-symbolic combined evaluation methods.