MMIR: A New Benchmark for Consistency Reasoning Capabilities of Multimodal Large Models

The UCSC research team released the MMIR (Multimodal Inconsistency Reasoning) benchmark, the first systematic framework dedicated to evaluating the reasoning ability of multimodal large language models (MLLMs) to detect image-text inconsistencies. This benchmark covers five reasoning-intensive inconsistency types, reveals significant shortcomings of current mainstream models in complex multimodal reasoning, and marks an important shift in multimodal model evaluation from 'being able to understand' to 'being able to judge'.

With the rapid development of MLLMs in tasks like image-text understanding and visual question answering, a core question emerges: Do models have deep reasoning capabilities to identify subtle inconsistencies between images and text? Current mainstream evaluation benchmarks mostly focus on correct image-text understanding and generation, but rarely touch on inconsistency detection and reasoning—which is crucial in real-world scenarios such as news verification, social media moderation, and legal document review.

The MMIR benchmark contains 534 test samples, covering five inconsistency types: factual contradiction, identity misattribution, context mismatch, quantity difference, and spatiotemporal incoherence. To ensure sample quality, the team adopted a four-stage filtering process: initial collection of multi-source candidate image-text pairs → manual annotation and classification of inconsistency types → multiple rounds of cross-validation and expert review to eliminate low-quality samples → grading based on reasoning complexity and depth of background knowledge.

Evaluation results show significant limitations of current models: In open-ended questions, the optimal o1 model achieves an overall accuracy of only 51.40% (as low as 38.73% for poster categories), while GPT-4o reaches 33.14%; open-source models are even weaker, with Qwen2.5-VL-7B (17.60%), LLaVA-NeXT-7B (14.70%), and InternVL2.5-8B (14.23%). Model performance slightly improves in multiple-choice settings (o1 reaches 52.15%, GPT-4o reaches 47.75%), but still falls far short of practical application requirements.

MMIR reveals key challenges: 1. Insufficient fine-grained visual understanding, making it hard to capture image details; 2. Cross-modal alignment bias, with imprecise matching between visual features and linguistic semantics; 3. Broken reasoning chains, poor maintenance of consistency in long logical chains; 4. Domain knowledge dependency, insufficient specific background knowledge affecting judgment. Continuous breakthroughs are needed in these directions in the future.

The industry needs to recognize the current capability boundaries of models and retain manual review in key scenarios like content moderation and fact-checking. Meanwhile, targeted fine-tuning using the MMIR dataset can improve models' inconsistency detection capabilities—this dataset has been open-sourced for community exploration.

The MMIR benchmark provides a standardized evaluation tool for the inconsistency reasoning capabilities of MLLMs. Current model performance indicates this field remains challenging; future breakthroughs are needed in fine-grained visual understanding, precise cross-modal alignment, and long-chain logical reasoning to achieve reliable multimodal intelligent systems.