ESPIRE Benchmark: Evaluating Embodied Spatial Reasoning Capabilities of Vision-Language Models

Current vision-language models (VLMs) perform well in tasks like image captioning and visual question answering, but they have shortcomings in embodied spatial reasoning (understanding and reasoning about physical spatial relationships). ESPIRE (Embodied Spatial Reasoning Benchmark) is a diagnostic benchmark targeting this capability. Through designs such as embodied perspectives, multi-level spatial relationships, and diverse reasoning types, it evaluates models' spatial understanding abilities, reveals their limitations, and provides directions for improvement.

Embodied intelligence emphasizes the interaction between agents and the physical environment, with spatial reasoning as its core. Humans can easily understand spatial relationships, but existing VLMs perform poorly. The motivations for creating ESPIRE are:

1. Traditional benchmarks (e.g., ImageNet, COCO) lack in-depth examination of spatial relationships;
2. Applications like robotics and autonomous driving require real spatial understanding capabilities;
3. There is a need for fine-grained testing to identify model capability defects.

ESPIRE's design revolves around three core principles:

1. Embodied Perspective: First-person scene descriptions that simulate an agent's actual observations (perspective-dependent, partially observable, dynamically updated);
2. Multi-level Spatial Relationships: Covers basic topology (inside/outside, adjacent), orientation (front/back/left/right), distance, composite (multi-object configuration), and functional (support, containment) relationships;
3. Diverse Reasoning Types: Includes tasks like description, prediction, planning, and counterfactual reasoning.

ESPIRE uses a procedural approach to build its dataset:

1. Scene Generation: Construct indoor scenes based on simulation engines, using 3D models like furniture to ensure physical plausibility;
2. Annotation Strategy: Automatic spatial relationship extraction + manual verification + expert review;
3. Question Generation: Fill natural language questions using templates, design distractors, and grade by difficulty.

ESPIRE evaluations reveal the limitations of VLMs:

• Fragile Relational Reasoning: Low accuracy in complex multi-object relationships (close to random for three or more objects);
• Perspective Sensitivity: Contradictory judgments under different perspectives, lacking stable spatial representations;
• Language-Visual Misalignment: Can describe visual content but poorly matches language with scenes;
• Difficulty in Compositional Generalization: Limited generalization ability for novel spatial configurations. Evaluation dimensions include perceptual accuracy, relationship understanding, compositional reasoning, etc. Error types include hallucinations, perspective confusion, relationship reversal, etc.

ESPIRE provides directions for VLM improvement:

1. Architecture Level: Introduce explicit spatial representation modules, geometric reasoning layers, and multi-perspective fusion;
2. Training Strategy: Data augmentation, curriculum learning (from simple to complex), contrastive learning;
3. Evaluation Practice: Incorporate into standard evaluation processes, error case analysis, human-machine comparison.

Application Value:

• Academic: Standardized evaluation tool, capability diagnosis framework, progress tracking;
• Industry: Reference for model selection, understanding of capability boundaries, guidance for improvement directions;
• Education: AI capability demonstration, cognitive science comparison, public communication. Future Directions: Expand to dynamic scenes, real-world data, multi-modal inputs, interactive evaluation, cross-cultural studies. The open-source codebase supports reproducibility, scalability, and collaborative improvement.