Geometric Structure of Arithmetic Ability in Large Language Models: An Analysis of the Shape-of-Addition Study

The ICML 2026 paper Shape-of-Addition reveals the intrinsic mechanism of arithmetic ability in large language models by analyzing the geometric structure of residual streams during multi-operand addition, and discovers a key geometric pattern called Iso-Raw-Sum Trajectory (IRST).

• Original Author/Maintainer: RL-MIND
• Source Platform: GitHub
• Original Title: Shape-of-Addition
• Original Link: https://github.com/RL-MIND/Shape-of-Addition
• Source Publication/Update Date: 2026-05-29

Large Language Models (LLMs) have demonstrated remarkable capabilities in various natural language processing tasks, but they exhibit puzzling fragility in basic arithmetic operations. This contradictory phenomenon suggests a gap between the model's internal computation mechanism and its discrete output. Why can a model that generates fluent prose and writes complex code frequently make mistakes in simple addition?

Traditional research often treats LLMs as black boxes, inferring their internal mechanisms through input-output behavior analysis. However, this approach is difficult to reveal the true internal representations of models when processing arithmetic operations. The RL-MIND team's research adopts a different approach: by analyzing the geometric structure of the residual stream when the model performs multi-operand addition, they attempt to understand the arithmetic ability of LLMs from an internal perspective.

The core finding of the research team is a geometric structure called Iso-Raw-Sum Trajectory (IRST). This discovery reveals that the internal representation space of LLMs follows a specific geometric trajectory when performing addition operations.

In the Transformer architecture, the residual stream refers to the path through which information is transmitted from the input layer to the output layer. The output of each layer is added to the input (residual connection) to form an information flow. By analyzing the geometric properties of this stream, researchers can observe changes in the model's internal state when processing specific tasks.

The study found that during multi-operand addition, the model's residual stream exhibits the following features:

1. Iso-Sum Trajectory: Inputs with the same raw sum move along similar trajectories in the residual space
2. Geometric Consistency: Representations across different layers maintain consistency in arithmetic structure
3. Layer-wise Evolution: As the number of layers increases, arithmetic representations gradually transform from implicit to explicit

The research team designed a series of carefully controlled experiments to explore the arithmetic mechanism of LLMs:

1. Multi-operand Addition Task: Test the model's ability to handle addition with different numbers of operands
2. Residual Stream Tracking: Track activation patterns of specific layers and neurons through intervention analysis
3. Geometric Analysis: Visualize high-dimensional representation spaces using dimensionality reduction techniques (e.g., PCA, t-SNE)
4. Causal Intervention: Verify the functional role of specific components by modifying intermediate layer representations

The project provides complete data processing and analysis code, including:

• Data Generation Module: Create standardized arithmetic test datasets
• Model Hooks: Used to extract and analyze intermediate layer representations
• Geometric Analysis Tools: Calculate trajectory similarity, subspace projection, etc.
• Visualization Scripts: Generate various charts in the paper