ReactBench: A Causality-Driven Evaluation Benchmark for Systematically Diagnosing the Root Causes of Multimodal Hallucinations

ReactBench is a groundbreaking multimodal hallucination evaluation benchmark that, for the first time, assesses the hallucination issues of multimodal large language models (MLLMs) from a causality-driven perspective rather than a simple result-detection approach. It addresses the pain points of existing benchmarks—focusing only on hallucination results, using simplified scenarios, and failing to challenge state-of-the-art models—by adopting a multi-task design and exam-style evaluation format to systematically expose and diagnose the causes of hallucinations. Its core components include four targeted tasks and a chain-of-thought (CoT) reasoning diagnosis method. Experiments reveal the vulnerability of current models, which is of great significance to the development of multimodal AI.

Multimodal large language models (MLLMs) have made rapid progress in the field of vision-language understanding, but their core issue is the tendency to generate hallucinations inconsistent with visual inputs. Most existing evaluation benchmarks only focus on detecting hallucination results and rarely explore the root causes; moreover, they rely on simplified scenarios and limited evaluation formats, failing to pose a real challenge to state-of-the-art models.

ReactBench designs four targeted tasks, each addressing a specific cause of hallucinations:

1. Relation Erasure: Modify the spatial configuration of objects (position, occlusion) to test spatial relationship understanding and expose co-occurrence biases;
2. Counterfactual Attributes: Modify object attributes (color, shape) to create counterfactual scenarios, testing the balance between visual perception and linguistic knowledge and exposing linguistic priors;
3. Change Tracking: Require comparing two images to identify changes, testing cross-image comparison ability and exposing cross-image comparison perception defects;
4. Dense Counting: Test the ability to count high-density similar objects, exposing fine-grained perception bottlenecks.

ReactBench adopts chain-of-thought (CoT) reasoning diagnosis, going beyond traditional accuracy evaluation. Its advantages include:

• Interpretability: Analyze the reasoning process to identify biased steps;
• Precise Localization: Know where and why the model went wrong;
• Guided Improvement: Targeted optimization of model architecture or training strategies.

ReactBench evaluations show that current MLLMs are still significantly vulnerable to specific hallucination triggers—even models that perform well in standard evaluations expose serious weaknesses. Practical implications:

• Model Selection: Need to focus on performance in specific hallucination types;
• Safety Assessment: Comprehensive diagnosis is required before critical applications (medical, autonomous driving);
• Continuous Improvement: Provide a reproducible and extensible platform to support model iteration.

ReactBench marks a new stage in multimodal hallucination research (shifting from detection to understanding), which is crucial for building reliable and interpretable systems:

• Researchers: Provide a systematic experimental platform to explore the impact of architecture/training strategies on hallucinations;
• Industry: Offer new tools for model evaluation and quality assurance;
• Users: Future products will be more reliable and have fewer hallucinations.

ReactBench is not only an evaluation benchmark but also a methodological innovation in the multimodal AI field. It systematically diagnoses hallucinations from a causal perspective, paving the way for building robust and trustworthy MLLMs. The project has been open-sourced; researchers and developers can visit the ReactBench homepage to get information and use it.