GeoWeaver: Endowing Visual Tokens with Spatial Awareness Using Geometric Evidence Before Scene Reasoning

Multimodal large language models (MLLMs) have made significant progress in visual understanding, but their performance in spatial reasoning tasks is subpar. GeoWeaver proposes a pre-reasoning geometric grounding framework that fundamentally addresses the insufficient geometric understanding of MLLMs by adaptively assigning the most relevant geometric abstractions to each visual token. The core idea is to treat geometric information as a premise for representation rather than an auxiliary signal for late fusion.

Current methods for incorporating geometric information into MLLMs include structural branches, 3D perception supervision, fusion during reasoning, and long-term memory. However, they share a common problem: coarse-grained processing—treating geometric cues as a unified signal shared by all visual tokens, ignoring the different role requirements of different tokens in spatial scenes (e.g., foreground objects need precise shapes, backgrounds need layout depth), leading to information redundancy or loss.

The core of GeoWeaver is pre-reasoning geometric grounding:

1. Multi-level geometric evidence library: Build an evidence library containing low-level (edges, corners), mid-level (planes, curved surfaces), and high-level (object shapes, scene layouts) geometric representations;
2. Token-adaptive retrieval: Analyze the spatial role of tokens (foreground/background/transition), select matching geometric representations from the library, and dynamically assign weights;
3. Residual grounding operation: Incorporate geometric evidence via residual connections, preserving original semantics while allowing learnable fusion strength.

Experiments cover tasks such as spatial relationship understanding, navigation path planning, physical reasoning, and temporal spatial reasoning. The results show:

1. Outperforms baseline models on all spatial reasoning benchmarks;
2. Maintains competitiveness in general visual-language tasks;
3. Compatible with different MLLM architectures, with good transferability.

GeoWeaver redefines the role of geometric information: from an auxiliary signal for late fusion to a premise for representation (shaping the foundation of visual representation before reasoning). Insights for architecture design:

• Visual encoders need to retain geometric information simultaneously;
• Different visual features require differentiated processing;
• Representation quality determines the upper limit of reasoning.

Application prospects include robot visual navigation, AR/VR spatial anchoring, and anatomical structure understanding in medical imaging. The research team commits to open-sourcing the code and pre-trained models. Repository address: https://github.com/yahooo-m/GeoWeaver, supporting reproduction, task expansion, and technical integration.

The core insight of GeoWeaver is that different visual tokens require different geometric evidence. It enhances spatial reasoning capabilities through pre-reasoning grounding and proposes a new paradigm: geometric information shapes representation first rather than being a remedy during reasoning. Future directions include exploring richer geometric abstractions, integrating 3D representation learning, and validating more real-world scenarios.