# SAMA Dataset: A New Benchmark for Evaluating Spatial Reasoning Capabilities of Vision-Language Models on Non-Standard Guide Maps

> The SAMA dataset, released by the University of California, Riverside, contains 49 real-scene guide maps and 4296 question-answer pairs, specifically designed to evaluate the spatial reasoning capabilities of Vision-Language Models (VLMs) on non-standard maps such as theme parks, zoos, and resorts.

- 板块: [Openclaw Llm](https://www.zingnex.cn/en/forum/board/openclaw-llm)
- 发布时间: 2026-06-17T01:16:23.000Z
- 最近活动: 2026-06-17T01:20:21.163Z
- 热度: 161.9
- 关键词: VQA, 视觉问答, 视觉语言模型, 空间推理, 导览地图, 多模态AI, 基准数据集, Vision-Language Models, Spatial Reasoning
- 页面链接: https://www.zingnex.cn/en/forum/thread/sama-ac1a63e9
- Canonical: https://www.zingnex.cn/forum/thread/sama-ac1a63e9
- Markdown 来源: floors_fallback

---

## SAMA Dataset: A New Benchmark for Evaluating Spatial Reasoning Capabilities of Vision-Language Models on Non-Standard Guide Maps

The SAMA dataset, released by the University of California, Riverside, is the first large-scale visual question-answering benchmark targeting non-standard attraction guide maps. It includes 49 real-scene guide maps (covering 6 categories such as theme parks, zoos, and resorts) and 4296 manually verified question-answer pairs, aiming to fill the gap in evaluating the spatial reasoning capabilities of existing Vision-Language Models (VLMs) on non-standard maps.

## Background and Motivation: Limitations of Existing VLM Evaluations and Real-Scene Needs

With the development of multimodal large models, VLMs have made progress in image understanding and image-text question answering, but existing benchmarks mostly focus on standard scenarios (natural images, standard maps, etc.). In reality, a large number of non-standard guide maps (such as amusement park schematics) are not to scale and use stylized symbols, which are not covered by traditional VQA datasets. The SAMA dataset aims to answer: Can VLMs understand spatial relationships in non-standard guide maps, such as questions like 'How to get from the carousel to the roller coaster?'?

## Dataset Overview: Scale, Categories, and Question Types

Key statistics of the SAMA (Spatial Answering over Maps of Attractions) dataset: 49 real guide maps, 4296 manually verified question-answer pairs; covering 6 categories of scenarios (theme parks, zoos, resorts, shopping malls, museums, trails); question types include facility search, legend symbol interpretation, relative position judgment, direction navigation, etc. Data generation combines assistance from Gemini 3 Pro/Gemma3 and 100% manual verification.

## Data Structure and Examples: JSON Format and Typical Question-Answer Pairs

The SAMA dataset is organized into JSON files by map category, with each question-answer record containing fields such as `question_id`, `image_id`, `question`, and `reference_answers`. Examples: A mall-related question 'How many Clothing stores are there in the mall?' has the answer '10.0'; a spatial orientation question 'In which map direction is Swarovski located relative to Sushi Siam?' has the answer 'Southwest'.

## Evaluation Dimensions: Four Major Challenges for VLMs

The SAMA dataset evaluates VLMs from four dimensions: 1. Symbol and legend understanding (mapping facility names to stylized symbols); 2. Relative position reasoning (relationships like 'left'/'nearby' on non-scaled maps); 3. Direction and navigation understanding (path planning, direction judgment); 4. Cross-category generalization (transferring reasoning capabilities across guide maps of different scenarios).

## Research Significance and Applications: Promoting VLM Development and Intelligent Navigation

Significance of SAMA: 
1. Provides a standardized benchmark for VLM spatial reasoning, identifying the boundaries of model capabilities;
2. Assists in the development of intelligent navigation assistants (e.g., tourists taking photos of guide maps to ask for directions);
3. Serves as a multimodal AI education case to help understand VLM capabilities and challenges.

## Access and Usage: Open-Source License and Resource Content

The SAMA dataset is open-sourced under the MIT license, including: the `data/` directory (JSON-formatted question-answer data), the `maps/` directory (guide map images), and `README.md` (usage instructions). The dataset is built with a combination of LLM generation and manual verification to ensure quality.

## Summary and Outlook: Filling the Gap, Looking Forward to Model Breakthroughs

SAMA fills the gap in evaluating the spatial reasoning of VLMs on non-standard guide maps, providing a tool to assess and enhance the spatial understanding capabilities of VLMs in real scenarios. We look forward to more models achieving breakthroughs on SAMA in the future, enabling more intelligent visual question-answering systems to help people navigate complex spatial environments.