Panoramic Guide to Spatial and 3D World Models: From Cognitive Maps to Embodied Intelligence

Original Author/Maintainer: Masoud Jafaripour Source Platform: GitHub Original Title: Awesome-Spatial-and-3D-World-Models Original Link: https://github.com/Masoudjafaripour/Awesome-Spatial-and-3D-World-Models Release Time: June 14, 2026

This article introduces an open-source library that systematically organizes research resources on spatial and 3D world models, covering core directions such as spatial memory, cognitive maps, predictive reasoning, planning and decision-making, and embodied intelligence, providing researchers and developers with a complete technical map of this field.

One of the core features of human intelligence is the understanding and application of space, enabling navigation, prediction, planning, and other abilities based on an internal "world model". Traditional AI systems perform clumsily in spatial tasks and lack an internal understanding of the world's structure. Research on spatial and 3D world models is endowing machines with human-like spatial cognitive abilities, providing key components for the development of robotics and general AI.

The Awesome resource library maintained by Masoud Jafaripour systematically organizes papers, datasets, benchmarks, and open-source code in this field, adopting a problem-oriented classification system:

1. Spatial World Models: Topological representation (node connections), metric representation (precise geometry), hybrid representation (hierarchical architecture);
2. 3D World Models: Explicit representation (voxels/point clouds), implicit representation (NeRF/occupancy networks), semantic 3D representation (geometry + semantics);
3. Video World Models: Autoregressive models, diffusion models, combination of world models and controllers;
4. Physical World Models: Physics engine-based models, learning-based physical models.

The core capabilities of world models include:

• Spatial Memory: Storing/recalling spatial experiences, addressing challenges such as limited storage and partial observability; the resource library includes grid/graph/end-to-end memory networks;
• Cognitive Maps: Abstracting the spatial structure of the environment, encoding positional relationships and path attributes, etc., which requires solving problems like perception extraction and uncertainty handling;
• Prediction and Reasoning: Forward prediction (environment evolution), reverse reasoning (cause inference), counterfactual reasoning (strategy evaluation);
• Planning and Decision-Making: Model-based reinforcement learning (e.g., MuZero), hierarchical planning (combination of high and low levels).

Embodied intelligence learns and reasons through physical interaction, and the world model is a core component:

• Vision-Language-Action Models: Integrating vision, language, and action control (e.g., RT-2, PaLM-E), which requires solving problems like multi-modal alignment and instruction ambiguity;
• Simulation-to-Real Transfer: Transferring from simulation training to real robots, facing domain difference challenges; the resource library includes technologies such as domain randomization and adaptation.

The resource library organizes key datasets and benchmarks:

• Indoor Scenes: Matterport3D, ScanNet (3D scanning data);
• Robotic Manipulation: RLBench, CALVIN (manipulation task data);
• Navigation Benchmarks: Habitat, iGibson (simulation environments and evaluation protocols).

Application prospects include robotics (environment understanding/planning), autonomous driving (safe decision-making), and virtual reality (immersive experience). However, there are still challenges: generalizable world models, open-world complexity, model safety and interpretability, which need to be addressed through interdisciplinary cooperation.

Research on spatial and 3D world models is a window to understanding the essence of intelligence. Human intelligence relies on understanding the physical world, and AI also needs to develop internal world models. This resource library provides an entry point for researchers. With technological progress, world models will become standard components of AI, paving the way for general AI.