Beyond Cartesian Illusion: A New Test of Spatial Theory of Mind for Multimodal Large Models

This article focuses on the "Cartesian Illusion" problem in embodied spatial intelligence of multimodal large language models (MLLMs) — over-reliance on text probability distributions and lack of understanding of the 3D topological structure of the physical world. It proposes a cognitive-perceptual bottleneck module and an anchor-based embodied spatial decomposition chain of thought, aiming to improve the models' performance in second-order theory of mind and embodied intelligence tasks. This research has important implications for the cognitive paradigm of embodied AI and fields such as autonomous driving and robot collaboration.

Multimodal large language models perform well in general reasoning tasks, but have fundamental limitations in embodied spatial intelligence — the "Cartesian Illusion": over-reliance on text probability distributions and lack of understanding of the 3D topological structure of the physical world. This limitation is particularly prominent in multi-agent scenarios, where agents need to have second-order theory of mind (inferring others' beliefs about the environment, influenced by their physical orientation and perceptual constraints).

This article explores the limits of MLLMs through a novel audio-visual task: predicting agent B's estimation of agent A's relative position. This task includes three major challenges: 1. Perspective transformation (understanding space from agent B's perspective); 2. Modeling of perceptual constraints (judging whether agent A is within agent B's field of view); 3. Modality fusion (dynamically balancing audio and visual information). The zero-shot test accuracy of existing MLLMs is only 42%, exposing fundamental difficulties in understanding spatial symmetry and out-of-view ambiguities.

The research proposes a cognitive-perceptual bottleneck module that abandons rigid coordinate transformations and adopts flexible reasoning methods. The core innovation is the anchor-based embodied spatial decomposition chain of thought:

1. Establish a local coordinate system: Build a self-centered coordinate system for agent B to understand the "egocentric" perspective;
2. Dynamic modality weighting: Adjust the weights of audio and visual information based on whether agent A is within agent B's visual cone (visual information dominates when in view, while auditory information is key when out of view), realizing the "geometry-to-semantics" projection and promoting deep spatial understanding.

The experiments reveal key findings:

1. Limitations of pure egocentric/allocentric baselines: The former lacks understanding of others' perspectives, while the latter ignores the physical constraints of the perceiving subject;
2. Robustness of the sensory-constrained chain of thought: Compared to baselines, the new method improves accuracy and provides an interpretable and controllable reasoning framework;
3. Difficulties in understanding spatial symmetry: Existing models have systematic problems in handling left-right mirroring and out-of-view ambiguities, reflecting a lack of 3D spatial representation capabilities.

Technical Significance:

• Cognitive-perceptual reasoning: Perception is part of cognition, and we need to understand the boundaries of "knowing" (perceptual constraints);
• Modality-perceptual reasoning: Dynamically balance information from different modalities instead of simple feature concatenation;
• Embodied foundation of theory of mind: ToM is rooted in embodied experience, and we need to consider others' physical existence and perceptual constraints. Application Prospects: Autonomous driving (understanding the perspectives of traffic participants), robot collaboration (mutual modeling among multiple robots), VR/AR (spatial reasoning and perspective transformation), and assistive technologies (navigation for the visually impaired considering perceptual constraints).

Limitations and Future Directions:

1. 3D spatial representation: Models need to truly understand 3D space instead of 2D projections;
2. Dynamic scene reasoning: Extend to dynamic environments;
3. Cross-modal spatial learning: Unify multi-modal spatial representations. Conclusion: MLLMs need to go beyond text probability reasoning and establish the ability to understand the spatial structure of the physical world. This research takes an important step toward embodied intelligence through the cognitive-perceptual bottleneck and anchor-based spatial decomposition, and calls for the field to break through the Cartesian Illusion.