SAMA Dataset: A New Benchmark for Evaluating Spatial Reasoning Capabilities of Vision-Language Models on Non-Standard Guide Maps

The SAMA dataset, released by the University of California, Riverside, is the first large-scale visual question-answering benchmark targeting non-standard attraction guide maps. It includes 49 real-scene guide maps (covering 6 categories such as theme parks, zoos, and resorts) and 4296 manually verified question-answer pairs, aiming to fill the gap in evaluating the spatial reasoning capabilities of existing Vision-Language Models (VLMs) on non-standard maps.

With the development of multimodal large models, VLMs have made progress in image understanding and image-text question answering, but existing benchmarks mostly focus on standard scenarios (natural images, standard maps, etc.). In reality, a large number of non-standard guide maps (such as amusement park schematics) are not to scale and use stylized symbols, which are not covered by traditional VQA datasets. The SAMA dataset aims to answer: Can VLMs understand spatial relationships in non-standard guide maps, such as questions like 'How to get from the carousel to the roller coaster?'?

Key statistics of the SAMA (Spatial Answering over Maps of Attractions) dataset: 49 real guide maps, 4296 manually verified question-answer pairs; covering 6 categories of scenarios (theme parks, zoos, resorts, shopping malls, museums, trails); question types include facility search, legend symbol interpretation, relative position judgment, direction navigation, etc. Data generation combines assistance from Gemini 3 Pro/Gemma3 and 100% manual verification.

The SAMA dataset is organized into JSON files by map category, with each question-answer record containing fields such as question_id, image_id, question, and reference_answers. Examples: A mall-related question 'How many Clothing stores are there in the mall?' has the answer '10.0'; a spatial orientation question 'In which map direction is Swarovski located relative to Sushi Siam?' has the answer 'Southwest'.

The SAMA dataset evaluates VLMs from four dimensions: 1. Symbol and legend understanding (mapping facility names to stylized symbols); 2. Relative position reasoning (relationships like 'left'/'nearby' on non-scaled maps); 3. Direction and navigation understanding (path planning, direction judgment); 4. Cross-category generalization (transferring reasoning capabilities across guide maps of different scenarios).

Significance of SAMA:

1. Provides a standardized benchmark for VLM spatial reasoning, identifying the boundaries of model capabilities;
2. Assists in the development of intelligent navigation assistants (e.g., tourists taking photos of guide maps to ask for directions);
3. Serves as a multimodal AI education case to help understand VLM capabilities and challenges.

The SAMA dataset is open-sourced under the MIT license, including: the data/ directory (JSON-formatted question-answer data), the maps/ directory (guide map images), and README.md (usage instructions). The dataset is built with a combination of LLM generation and manual verification to ensure quality.

SAMA fills the gap in evaluating the spatial reasoning of VLMs on non-standard guide maps, providing a tool to assess and enhance the spatial understanding capabilities of VLMs in real scenarios. We look forward to more models achieving breakthroughs on SAMA in the future, enabling more intelligent visual question-answering systems to help people navigate complex spatial environments.