GeoR-Bench: A New Benchmark for Evaluating Multimodal Models' Visual Reasoning Capabilities in Earth Sciences

Institutions including the Chinese University of Hong Kong have released the GeoR-Bench benchmark, covering 440 samples, 6 earth science domains, and 24 task types. Tests show that top closed-source models achieve an accuracy of only 42.7%, while open-source models reach just 10.3%, revealing a severe bottleneck in current multimodal AI's reasoning capabilities in earth sciences.

With climate change, frequent natural disasters, and increasing pressure on environmental protection, there is an urgent need for intelligent systems that can understand and predict changes in the Earth system. Earth science intelligence has become one of the socially valuable application directions in the AI field. However, the performance of existing multimodal large language models on earth science reasoning tasks lacks systematic evaluation, and existing benchmarks mostly focus on specific scenarios, making it difficult to reflect open-ended earth science problems in the real world.

GeoR-Bench is a benchmark specifically designed for earth science visual reasoning, combining reasoning with visual editing tasks. It contains 440 samples, covering 6 core earth science categories: atmospheric science, hydrological science, geological science, ecological science, agricultural science, and human geography, subdivided into 24 task types, including satellite remote sensing images, maps, scientific charts, and other forms.

GeoR-Bench establishes a three-dimensional evaluation framework: 1. Reasoning ability: Evaluate understanding of logical chains, such as causal inference, temporal analysis, and spatial relationship reasoning; 2. Consistency: Check internal consistency between visual and scientific logic; 3. Quality: Evaluate the visual authenticity and scientific accuracy of generated images.

Tests on 21 mainstream multimodal models show: The overall strict accuracy of top closed-source models is only 42.7%, while the best-performing open-source model reaches just 10.3%. Common issues: Visual consistency and image quality are better than scientific accuracy, and models mostly perform surface pattern matching rather than understanding internal mechanisms.

Earth science reasoning faces three major challenges: 1. Demand for cross-spatiotemporal long-range reasoning; 2. Highly professional and domain-specific data; 3. Need to integrate multi-source heterogeneous information for comprehensive judgment.

Future model development should focus on deep understanding of earth science principles: Introduce more scientific literature, textbook knowledge, and expert annotations; Incorporate domain-specific inductive biases; Adopt stricter scientific accuracy standards. GeoR-Bench provides researchers with an evaluation tool and points out improvement goals for developers.

GeoR-Bench fills the gap in earth science AI evaluation; it is not only a testing tool but also reflects the gap between current technology and usable earth science intelligence. In today's era of urgent climate change and environmental protection, narrowing this gap has important practical significance.