GGBench: A Geometric Generation and Reasoning Benchmark for Unified Multimodal Models

GGBench is a geometric generation and reasoning benchmark designed specifically for unified multimodal models (UMMs). It is the first to integrate discriminative understanding and controlled image generation capabilities into a single evaluation framework. Through geometric construction tasks, it tests whether models can fuse language comprehension with precise visual construction abilities. It covers a multi-dimensional evaluation system, reveals the shortcomings of current models in cross-modal alignment and other aspects, and provides open-source datasets and evaluation tools for the research community to promote the development of the multimodal AI field.

In recent years, unified multimodal models have made significant progress in visual understanding and text generation. However, existing evaluation methods often test discriminative understanding and unconstrained image generation separately, making it difficult to fully measure the real ability of models in complex reasoning tasks involving precise visual construction. Against this background, GGBench emerged, integrating the evaluation of language comprehension and precise visual construction abilities to provide a systematic testing platform for the generative reasoning capabilities of UMMs.

Ideal Test Scenario: Geometric Construction

The reasons geometric construction becomes an ideal test scenario: 1. It has clear logical structure and mathematical rigor, requiring understanding of language and generating graphics that conform to theorems; 2. It involves multiple reasoning steps, showing the chain of thought; 3. Correctness can be objectively verified through mathematical rules. GGBench contains 1411 geometric construction problems, covering 8 categories such as basic construction, circle properties, geometric transformations, etc., to ensure comprehensive evaluation.

Multi-Dimensional Evaluation System

• VLM-T: Text reasoning evaluation (1-5 points), examining the logic and clarity of problem-solving steps;
• VLM-I-Mid: Intermediate process image evaluation, focusing on step accuracy, consistency, and problem-solution matching;
• VLM-I-Res: Final result image evaluation (1-5 points), measuring geometric accuracy, annotation clarity, and consistency;
• Image quality metrics: Objective pixel-level evaluations such as LPIPS, PSNR, SSIM.

Research Findings

1. Current models perform far from ideal in geometric generation and reasoning tasks; even the best models face significant difficulties in complex problems;
2. Models perform better in the planning phase than in the execution phase—they can generate reasonable steps but have obvious deviations when converting to visual construction;
3. Models show large differences in ability across different geometric problem types: basic construction is easy, while complex theorem application and trajectory construction are extremely challenging.

Typical Cases

• Success cases: When the problem structure is clear, concepts are basic, and steps are limited, models can accurately parse the problem, formulate strategies, and generate standard graphics;
• Failure cases: Common issues include misunderstanding problem requirements, ignoring key constraints, accumulating errors in intermediate steps, and generating "hallucinatory" elements that violate theorems.

GGBench reveals the limitations of current UMMs in precise visual generation tasks and emphasizes the importance of cross-modal alignment (needing to establish an accurate correspondence between language comprehension and image generation). Its multi-dimensional evaluation method can accurately locate model defects and provide directions for improvement. In addition, the GGBench team has open-sourced the dataset (available on Hugging Face) and evaluation tools, supporting automatic completion of comprehensive evaluations and providing valuable resources for the community.

GGBench marks a new stage in multimodal model evaluation. Future research can delve into: 1. Developing model architectures targeted at geometric reasoning; 2. Exploring more effective cross-modal alignment methods; 3. Extending the evaluation framework to other precise visual construction fields. More importantly, the multi-dimensional evaluation concept advocated by GGBench is expected to be promoted to a wide range of multimodal tasks, setting a benchmark for model capability evaluation in real-world applications and driving progress in the entire multimodal AI field.