MM-CoT: A Benchmark for Evaluating Visual Chain-of-Thought Reasoning Capabilities of Multimodal Models

MM-CoT is a benchmark dataset dedicated to evaluating the visual chain-of-thought reasoning capabilities of large multimodal language models. Addressing the limitation of traditional visual evaluation which only focuses on recognition results, it reveals the capabilities and limitations of current models in complex visual reasoning by forcing the display of reasoning processes, providing a key evaluation tool and improvement direction for the development of multimodal AI.

In recent years, multimodal models such as GPT-4V and Claude 3 have acquired visual understanding capabilities, but traditional benchmarks (e.g., ImageNet) only test object recognition and cannot evaluate complex visual reasoning (scene relationships, chart interpretation, cross-modal causality, etc.). Humans rely on chain-of-thought to solve visual problems—do multimodal models have this capability? MM-CoT was thus born.

MM-CoT is a benchmark for probing the visual chain-of-thought reasoning capabilities of multimodal models, forcing the display of reasoning processes. Visual chain-of-thought requires observing details, establishing relationships, cross-modal integration, and step-by-step reasoning. Its value lies in interpretability, error diagnosis, capability boundary localization, human-machine collaboration optimization, etc.

Task types cover visual logic puzzles, chart interpretation, scene causal reasoning, visual mathematics, and multi-image sequence reasoning; data construction uses manual annotation verification, diverse sources, difficulty grading, and adversarial design; evaluation metrics include final answer accuracy, reasoning process quality, visual grounding, and step completeness.

Current status: Strong text chain-of-thought capabilities but weak visual reasoning; marginal effects exist in model scale improvement; each model has domain expertise; hallucinations and over-reasoning exist. Failure modes: Surface pattern matching, text bias dominance, broken reasoning chains, insufficient fine-grained visual understanding.

Guiding model improvement (targeted strengthening of weak links); promoting benchmark evolution (from answer correctness to reasoning quality); application insights (identifying competent scenarios and scenarios requiring human supervision).

Benchmark evolution: More complex reasoning tasks, video understanding, real-world application scenarios, automated evaluation tools. Technological progress: Generating clear reasoning processes, integrating visual attention, training models to acknowledge limitations.

MM-CoT is an important evaluation tool; its focus on reasoning processes reflects the emphasis on interpretability. Current models still have room for improvement in deep visual reasoning, and MM-CoT will guide models to break through and achieve human-like visual reasoning capabilities.