Geo-Benchmark: An Open-Source Benchmark Framework for Evaluating Large Language Models' Climate Prediction Capabilities

CliDyn team has open-sourced the geo_benchmark framework (MIT license, Python-based) to systematically assess large language models (LLMs) in global climate data prediction tasks. Key features include generating global geographic grids, integrating multi-source geospatial data, and quantifying model accuracy in temperature and precipitation predictions. The project is available on GitHub (https://github.com/CliDyn/geo_benchmark) and was released on June 15, 2026.

LLMs have expanded their capabilities to text generation, code writing, and complex reasoning, but their performance in precise scientific domains like climate science remains under-evaluated. Traditional LLM benchmarks focus on language understanding/generation, lacking standardized tools for scientific computing and geospatial reasoning. Geo-Benchmark fills this gap by providing a framework for assessing LLM performance in climate prediction tasks.

Geo-Benchmark is a specialized benchmark for evaluating LLMs' climate prediction performance. Its core workflow includes:

1. Generating global geographic grids (supporting various resolutions)
2. Identifying land coordinates using shapefile data
3. Enhancing location data via multi-source geospatial information integration
4. Querying LLMs for temperature and precipitation predictions
5. Analyzing and visualizing results against real observational data (e.g., ERA5 reanalysis data)

The framework's technical highlights:

• Geographic Grid Processing: Modular system for flexible grid resolution selection
• Data Integration: Combines high-resolution coastline data, DEM, population distribution, and ERA5 data as reference
• Efficiency: Batch query mechanism and distributed processing for large-scale global evaluations
• Analysis Tools: Spatial RMSE analysis, monthly trend comparison, climate scenario analysis, and regional performance contrast

Tech stack: Python with GeoPandas, xarray, matplotlib/cartopy, PyYAML

Geo-Benchmark uses multiple metrics to assess LLM performance:

• RMSE: Core numerical accuracy metric for comparing predicted vs. observed values (supports single/multi-variable calculations)
• Spatial Distribution Analysis: Maps prediction errors to identify geographic regions with systemic biases
• Time Series Analysis: Evaluates model understanding of seasonal/climate periodicity
• Model Comparison: Supports parallel assessment of multiple LLMs (e.g., GPT series, Ollama-deployed models) for performance contrast

The framework serves three main purposes:

1. Academic Research: Climate scientists can evaluate LLMs' climate knowledge and identify blind spots in specific regions/phenomena
2. Model Development: LLM developers can use iterative evaluations to optimize geospatial and scientific reasoning capabilities
3. Education: Helps students understand climate data complexity and AI limitations in scientific prediction

Current limitations:

• Data Dependency: Evaluation quality relies on input data quality/coverage
• Compute Resources: High-resolution global evaluations require significant computational power
• Model Interface: Primarily supports API-based closed-source models and local Ollama deployments

Future plans:

• Expand support for more climate variables (humidity, wind speed)
• Integrate real-time meteorological data streams
• Develop city-level fine-grained evaluation
• Establish a public model performance leaderboard

Geo-Benchmark represents an important step in extending AI benchmarks to professional scientific fields. By providing a standardized, open-source tool, it helps researchers understand LLMs' capabilities and limitations in climate prediction. As climate change urgency grows, such tools are critical for assessing AI's reliability in supporting climate decision-making.