GAP-MLLM: Activating 3D Spatial Perception Capabilities of Multimodal Large Language Models via Geometry-Aligned Pre-training

GAP-MLLM proposes a novel geometry-aligned pre-training method aimed at enhancing the 3D spatial perception and understanding capabilities of multimodal large language models, bridging the gap between 2D vision and 3D geometry.

Original Author/Maintainer: ZestfulJX Source Platform: GitHub Original Title: GAP-MLLM Original Link: https://github.com/ZestfulJX/GAP-MLLM Source Publication/Update Time: 2026-05-28T06:42:55Z

Current multimodal large language models (MLLMs) have made significant progress in understanding 2D images, but still face major challenges when processing 3D spatial information. Traditional vision-language pre-training methods mainly focus on image-text alignment, lacking explicit modeling of depth, geometric structure, and spatial relationships. This leads to poor performance of existing models in tasks requiring 3D reasoning, such as spatial navigation, object localization, and scene understanding.

The GAP-MLLM project was born to address this core issue. The research team recognizes that to enable multimodal models to truly understand the physical world, a geometry-aware pre-training mechanism must be introduced to allow the model to establish a mapping from 2D vision to 3D geometry.

The core innovation of GAP-MLLM lies in proposing a "Geometry-Aligned Pre-training" paradigm. The key idea of this method is to explicitly introduce geometric supervision signals during the pre-training phase, allowing the model to learn to associate visual features with 3D spatial structures.

3D Geometry Representation Learning

The project adopts a multi-level 3D geometry representation strategy:

• Low-level: Extract depth estimation and surface normal information
• Mid-level: Understand spatial relationships between objects (e.g., "on top of", "to the left of")
• High-level: Perform geometric reasoning for the entire scene

Cross-Modal Geometry Alignment

Three alignment mechanisms are designed:

1. Point Cloud-Image Alignment: Contrastive learning to understand the relationship between 2D projections and 3D coordinates of the same 3D point
2. Geometry-Language Alignment: Associate geometric descriptions (e.g., "cube") with visual features
3. Spatial Relation Alignment: Learn the correspondence between spatial relation language concepts and visual scenes

Pre-training Task Design

Includes the following specialized tasks:

• Depth Prediction Task: Predict depth maps from single images
• Camera Pose Estimation: Infer shooting angles and camera parameters
• 3D Object Reconstruction: Reconstruct 3D shapes of objects from 2D images
• Spatial QA: Answer visual questions requiring 3D reasoning

GAP-MLLM is extended based on mainstream multimodal architectures, including the following modules:

Visual Encoder: Uses Vision Transformer (ViT) as the base, outputting multi-scale feature representations to support geometric reasoning at different granularities.

Geometry Encoder: A dedicated geometric information encoding module that receives inputs such as depth maps and surface normal maps, and encodes them into representations compatible with visual features.

Cross-Modal Fusion Layer: A geometry-aware attention mechanism that allows visual features and geometric features to guide each other, adjusting attention according to geometric constraints.

Language Decoder: A standard autoregressive language model architecture, with inputs including visual features and fused geometric-visual joint representations.

The 3D spatial perception capabilities of GAP-MLLM bring new possibilities to multiple fields:

• Robot Navigation and Manipulation: Supports robot vision-language instruction execution tasks
• Augmented Reality (AR) and Virtual Reality (VR): Helps AR devices understand physical spaces
• Autonomous Driving: Assists in spatial reasoning for road scenes
• Intelligent Interior Design: Understands 3D information such as room layouts and furniture placement

Challenges faced during development and their solutions:

Data Scarcity: High-quality 3D-language aligned data is scarce. Solutions include using synthetic data, designing self-supervised pre-training tasks, and extracting geometric information from existing 2D-language data.

Computational Efficiency: 3D geometric computation is time-consuming. Mitigated through efficient geometry encoder design and progressive training strategies.

Generalization Ability: Needs to work stably across different scenarios. Achieved through data augmentation and domain randomization techniques.

GAP-MLLM represents an important step towards 3D world understanding for multimodal large models. Future directions include:

• Extending to video understanding, introducing temporal 3D reasoning
• Combining with embodied intelligence to support physical interaction tasks
• Exploring more efficient 3D representation methods (e.g., combining Neural Radiance Fields (NeRF) with language models)
• Developing larger-scale geometry-language pre-training datasets

GAP-MLLM effectively activates the 3D spatial perception capabilities of multimodal large language models through its innovative geometry-aligned pre-training method. This work not only pushes the technical boundaries of multimodal learning but also provides a new technical foundation for application scenarios requiring 3D understanding, such as robotics, AR/VR, and autonomous driving. With enhanced 3D perception capabilities, multimodal large models will better serve practical tasks related to physical world understanding.