Unveiling the Geometric Essence of Large Language Models' Arithmetic Ability: An Analysis of the Shape-of-Addition Study

Research Team: Nanjing University RL-MIND Research Team Conference Published: ICML 2026 Key Findings:

• Identified the geometric structure in LLM addition operations—Isometric Raw Sum Trajectory (IRST)
• Proposed a noise quantization model, interpreting arithmetic errors as "geometric slippage" Research Significance: Provides a new geometric perspective for understanding and improving LLMs' numerical reasoning capabilities Original Link: https://github.com/RL-MIND/Shape-of-Addition Publication Time: May 2026

Large language models (LLMs) perform excellently in complex tasks, but their basic arithmetic operations exhibit puzzling vulnerability. This paradox suggests a fundamental disconnect between the model's internal continuous representation space and discrete outputs. Traditional views attribute errors to insufficient training data or limitations of tokenization strategies, while this study proposes a deeper explanation: arithmetic errors stem from the quantization conflict between continuous representations and discrete outputs.

By analyzing the geometric structure of residual flows during addition, the research team discovered the Isometric Raw Sum Trajectory (IRST), whose core features are:

1. Semantic Number Anchoring: The model's internal representations are anchored on semantic numbers, establishing a numerical topological structure
2. Continuous Carry Fiber Modulation: There exists a fiber structure composed of continuous carry potential between semantic anchors, forming a smooth transition region
3. Tension Between Geometry and Discreteness: The inherent tension between continuous geometric representations and discrete numerical outputs leads to systematic errors

Based on the IRST discovery, the team proposed the Noise Quantization Model, viewing arithmetic errors as geometric slippage with core mechanisms:

1. Continuous Carry Potential: Carry operations exist as continuous variables in the representation space (transition zone from 0 to 1)
2. Quantization Threshold Boundary: When carry potential crosses the threshold, the corresponding number is output, and the threshold forms the decision boundary
3. Neural Noise Driving: Internal noise pushes the carry potential into the wrong quantization interval, leading to output errors
4. Predictable Error Patterns: The geometric structure makes certain numerical combinations more prone to slippage, explaining the systematic nature of errors

The study reveals Detector Versatility:

• Coexisting Signal Separation: Lightweight detectors can decouple parallel representations of real and hallucinated answers from a single activation vector
• Intervention Possibility: By adjusting the projection of activation vectors, the slipped representations can be pushed back to the correct quantization interval
• Correction Strategies: Developed multiple methods such as MLP detector number replacement, linear detector guidance, and dual-stream correction

The Geometric Consistency Check method can detect and correct errors in real time:

• Representation Consistency: Correct arithmetic operations should maintain specific geometric consistency (adjacent number representations follow predictable relationships)
• Anomaly Detection: Mark potential errors when representations deviate from the expected trajectory
• Intervention Correction: Project the deviated representations back to the correct trajectory without regenerating the entire answer Experimental results: Significantly improved multi-digit addition accuracy, providing a path for reliable numerical reasoning systems

Open-Source Implementation: The team open-sourced the complete codebase, including activation tracking generators, detector training and evaluation, error decomposition analysis, and visualization tools (UMAP/PCA, etc.) Research Significance:

1. The IRST structure may be universally present in LLMs' processing of discrete concepts
2. The geometric perspective opens a new direction for neural network interpretability
3. Provides a theoretical foundation for model editing and correction techniques
4. Helps design targeted training strategies (e.g., regularization terms that enhance geometric consistency)