AI-Benchmarks: An Evaluation Framework for Spatial Reasoning Capabilities of Large Language Models

waifuai/ai-benchmarks is an open-source evaluation suite specifically designed to assess the spatial reasoning capabilities of large language models (LLMs). It uses a gradient-based scoring mechanism, supports standardized testing of multiple models via OpenRouter, and generates comparable leaderboard data, aiming to fill the gap in traditional evaluations regarding spatial reasoning capability assessment.

With the widespread application of LLMs in various tasks, systematic evaluation of their reasoning capabilities has become a key issue. Traditional evaluations focus on language understanding or knowledge question-answering, while assessment of complex spatial relationship reasoning is relatively weak. Spatial reasoning involves the understanding and inference of concepts such as object position, direction, and relative distance, which are crucial for scenarios like robot decision-making, autonomous driving path planning, and intelligent assistant interaction. The waifuai/ai-benchmarks project emerged to fill this evaluation gap.

ai-benchmarks is an open-source evaluation suite whose core goal is to provide repeatable and comparable quantitative assessments of LLM spatial reasoning capabilities, supporting integration of the command-line interface (CLI) into CI pipelines or automation scripts. Its main features include:

1. Focus on spatial reasoning: tasks are specifically designed to test spatial relationship understanding;
2. Gradient-based scoring mechanism: gives scores based on the proximity of the answer to the ideal solution;
3. OpenRouter multi-model integration: supports evaluating multiple LLMs at once;
4. Standardized input/output format: ensures result comparability;
5. Supports generating structured data to build leaderboards.

Evaluation Task Design

Includes four types of tasks: relative position judgment, path planning and navigation, spatial transformation reasoning, and 3D space understanding.

Gradient Scoring System

Unlike binary scoring, it gives scores based on the proximity of the answer to the ideal solution. For example, in coordinate tasks, answers closer to the correct coordinates get higher scores, which can more accurately reflect model capabilities and track fine-tuning improvements.

OpenRouter Integration Architecture

Through the OpenRouter unified API gateway, it achieves model diversity (no need to configure multiple models separately), cost optimization (unified billing), and result standardization (eliminating interference from API differences).

1. Model Selection Decision: Provides objective references for applications involving spatial reasoning (e.g., smart home control, robot instruction understanding) to help developers compare candidate models;
2. Model Fine-Tuning Effect Validation: Quickly verifies whether fine-tuning improves spatial reasoning capabilities and establishes a baseline for before-and-after comparison;
3. Academic Research Benchmark: Serves as a standardized testing platform for new models/algorithms, and its open-source nature allows task customization.

The usage flow is as follows:

1. Configure Environment: Install dependencies and configure the OpenRouter API key;
2. Define Test Set: Select or customize spatial reasoning test cases (the project provides pre-built datasets);
3. Run Evaluation: Specify the models to be tested and the test set via CLI, then start the automated evaluation;
4. Analyze Results: View scoring reports and statistical summaries to identify model strengths and weaknesses;
5. Generate Leaderboard: Aggregate multiple results to generate a shareable performance leaderboard.

Limitations

1. Evaluation Scope: Focuses on discrete spatial relationships, with insufficient support for continuous space, dynamic scenarios, and multi-modal spatial understanding;
2. Task Diversity: Targeted tasks for specific vertical domains (e.g., medical image spatial analysis) need to be supplemented by the community;
3. Scoring Subjectivity: The distance definition in gradient scoring has subjectivity, and different scenarios have different needs.

Future Directions

Introduce multi-modal evaluation tasks (combining images), support complex dynamic scenario simulation, and decompose spatial reasoning into sub-capabilities (sense of direction, distance estimation, etc.).

ai-benchmarks is an open-source evaluation framework focused on LLM spatial reasoning capabilities. Through gradient scoring, multi-model integration, and standardized processes, it provides a practical tool for developers and researchers. Against the backdrop of spatial reasoning becoming a key capability for LLM applications, this project is of great value in promoting model improvement and application implementation, and it is worth including in the technical evaluation toolbox.