GS-QA: A Comprehensive Evaluation Benchmark for Geospatial Question Answering

GS-QA is a comprehensive evaluation benchmark for geospatial question answering, containing 2800 question-answer pairs, covering 28 question templates, supporting multi-source reasoning and various answer types, and providing a comprehensive framework for evaluating the geospatial reasoning capabilities of large language models (LLMs). This benchmark is built based on OpenStreetMap and Wikipedia data, addressing the limitations of existing benchmarks in terms of scale, diversity, and cross-data source reasoning tests.

With the breakthroughs of LLMs in general question answering tasks, geospatial question answering (which requires handling complex reasoning such as spatial relationships, coordinates, and distance calculations) has become a research hotspot, with great application value yet significant challenges. Existing geospatial question answering benchmarks have limitations such as insufficient number of questions, limited coverage of spatial predicates, single answer type, and lack of cross-data source reasoning test scenarios, which restrict the development of the field.

GS-QA has three core features: 1. Multi-dimensional spatial predicate support (including basic spatial relationships, directional predicates, orientation filtering, etc.); 2. Rich answer types (entity names, coordinates, distance values, direction descriptions, counts, area/length aggregation); 3. Multi-source reasoning challenges (requiring integration of OpenStreetMap geographic information and Wikipedia factual knowledge).

GS-QA adopts a comprehensive evaluation framework that combines traditional text question answering metrics with geospatial-specific metrics: text matching metrics (exact match, partial match), distance error (evaluating the spatial deviation between predicted positions and real answers), and angle error (for direction-related questions), accurately reflecting the real capabilities of models.

The research team implemented nine baseline methods (direct prompting, RAG, Text-to-SQL, etc.) based on GPT-4o, Claude Sonnet4.6, and Ministral-3. Findings: Good performance on simple tasks (basic spatial predicates + entity answers); significant drop in accuracy for complex reasoning (complex predicates, numerical results, cross-data sources); multi-source reasoning is the most challenging.

The data construction process of GS-QA is scalable, based on the OpenStreetMap open geographic database + Wikipedia knowledge, generating high-quality question-answer pairs through templating. Application value: Broad prospects in fields such as intelligent navigation, urban planning, disaster emergency response, and tourism recommendation, providing a reliable evaluation standard for technology research and development.

GS-QA marks a new stage in geospatial question answering research. Future key directions include complex spatial reasoning, numerical calculation accuracy, and multi-source knowledge fusion. For researchers, GS-QA is both an evaluation tool and a mirror of technical level; we look forward to more innovative results in cross-disciplinary fields.