Urban Airspace Embodied Navigation Benchmark: How Far Are Large Models From Human-Level Spatial Action Capabilities?

This paper aims to evaluate the spatial action capabilities of Large Multimodal Models (LMMs). By constructing a city airspace goal-oriented navigation dataset with 5037 samples, it systematically assesses 17 representative models, reveals the rapid deviation of current models after critical decision fork points, and explores four improvement directions. The study shows that although LMMs have preliminary spatial action capabilities, there is still a significant gap from human-level performance.

LMMs have achieved remarkable results in visual-language understanding tasks, but lack real spatial decision-making and action capabilities. Embodied intelligence is crucial for scenarios such as robotics and autonomous driving. Urban 3D airspace navigation is an extremely challenging test scenario, which comprehensively tests the model's 3D spatial understanding, complex environment perception, and long-range planning capabilities.

The research team spent 500 hours building a high-quality dataset with 5037 samples, based on real urban 3D models, covering diverse tasks, vertical actions, and rich semantic information. Seventeen models were evaluated, divided into four categories: non-inferential LMMs, inferential LMMs, agent-based methods, and Visual-Language-Action models (VLAs).

Navigation errors do not accumulate linearly; instead, they deviate rapidly after critical decision fork points. Model failure modes include insufficient geometric perception, difficulty in perspective understanding, lack of spatial imagination, and long-term memory defects. This indicates that the core challenges of spatial navigation lie in grasping the global structure and ensuring the accuracy of key decisions.

The study explores four improvement directions: 1. Enhanced geometric perception (adding geometric annotations); 2. Cross-perspective understanding (training cross-perspective scene correspondence); 3. Cultivation of spatial imagination (predicting future observations or action consequences); 4. Long-term memory mechanism (introducing explicit memory architecture).

Current models have potential but are still far from practical application; the core challenge lies in high-level decision-making rather than low-level control. The combination of the four improvement directions may produce synergistic effects. We look forward to more research to promote the development of embodied intelligence, ultimately enabling AI to act freely in complex 3D spaces.