FRIEDA: A Benchmark for Evaluating Multi-step Map Reasoning Capabilities of Vision-Language Models

FRIEDA is a benchmark for evaluating multi-step map reasoning capabilities of vision-language models (VLMs) accepted by ICLR 2026. It focuses on open-ended multi-step map reasoning tasks, covering spatial relationships such as topology (boundary, inclusion, etc.), metrics (distance), and directions (orientation). It requires models to perform cross-map multi-hop reasoning. This benchmark fills the gap in map reasoning capability evaluation for existing VLMs, providing two dataset versions: Direct (pure reasoning) and Contextual (map selection required). It supports the evaluation of various open-source/closed-source models, facilitating the improvement of models' spatial reasoning capabilities and cross-domain research.

Maps are important tools for spatial information understanding, but existing VLM benchmarks mostly focus on general visual question answering or document understanding, lacking systematic evaluation for map reasoning. Map understanding requires mastery of complex spatial relationships (topology, metrics, directions), so FRIEDA was created to evaluate the performance of VLMs on open-ended, multi-step map reasoning tasks.

FRIEDA is built based on real map resources (from fields like geology, urban planning, environmental assessment, etc.) and adopts a spatial relationship classification framework from GIS theory:

• Topological relations: Boundary, equality, intersection, inclusion (unchanged with scale)
• Metric relations: Distance (requires understanding of scale and coordinates)
• Directional relations: Absolute orientation (east/south, etc.), relative position (left/right, etc.) The problem design follows the principles of multi-hop reasoning (needing multi-step analysis) and cross-map association (integrating information from multiple maps).

FRIEDA provides two dataset versions:

• Direct version: Directly presents questions and maps to test pure reasoning ability
• Contextual version: Requires selecting the correct map first to test document retrieval and selection ability The evaluation framework supports open-source (Llama, Qwen-VL, etc.), closed-source (GPT-4V, Claude, etc.), and custom models. The process is concise (e.g., running evaluation via command line), generating model answers and evaluation result files, with built-in performance optimizations like Flash Attention.

Research Value: Fills the gap in map reasoning evaluation for VLMs, provides standardized tools; promotes the improvement of models' spatial reasoning capabilities; facilitates cross-disciplinary research in computer vision, NLP, and geographic information science. Application Scenarios: Guides intelligent map question-answering systems (public assistants, professional report generation, educational tutoring); enhances geographic information retrieval (optimization of RAG systems); provides model selection references for developers.

Technical Details: Provides environment configuration guides (dependency installation, PyTorch, Flash Attention); data can be obtained via Hugging Face Hub or Google Drive; API keys for closed-source models are managed via environment variables. Community Resources: Project homepage (visualization, leaderboard), Hugging Face dataset, arXiv paper; code is open-source, contributions are welcome (submitting results, improving tools, expanding datasets).

Current Limitations: Language is mainly English; map types focus on professional fields, with limited coverage of consumer navigation maps; reasoning steps are relatively limited. Future Directions: Expand multi-language support; introduce dynamic maps (temporal changes) and interactive maps; add more complex reasoning steps.

As the first systematic benchmark for evaluating the map reasoning capabilities of VLMs, FRIEDA defines evaluation dimensions and standards, and provides high-quality data and tools. It will accelerate research on AI's spatial knowledge understanding capabilities, facilitate the application of VLMs in map-related scenarios, and enable AI to better utilize human spatial knowledge.