Do Large Vision-Language Models Really Reason? Visual Puzzle Benchmarks Reveal the Truth

Large Vision-Language Models (LVLMs) perform well in multimodal tasks, but is it true reasoning or superficial pattern matching? A recent systematic review uses a family of visual puzzle benchmarks to provide a rigorous evaluation framework for answering this core controversy and deeply investigate their abstract reasoning capabilities.

Visual puzzles, relying on visual information, clear constraint structures, verifiable solutions, and reducing dependence on external knowledge, have become a touchstone for testing LVLMs' abilities such as abstract reasoning and rule induction. Formally defined as a triple ⟨I, R, S⟩: I is the visual input, R is the rule constraint, and S is the structured solution space, which can precisely control task complexity.

The study uses multiple types of visual puzzle benchmarks: inductive reasoning (Raven's Progressive Matrices, procedurally generated matrices, ARC series), analogical reasoning (Bongard Problems, REBUS, etc.), algorithmic and deductive reasoning (procedural thinking, logical deduction), and geometric spatial reasoning (mental rotation, perspective projection, etc.), covering all reasoning dimensions comprehensively.

LVLMs show vulnerable performance in inductive tasks (e.g., RPM, ARC): performance drops sharply when there is distribution shift, they rely on superficial cues rather than abstract rules, perceptual limitations are intertwined with reasoning errors, and fluent language descriptions do not guarantee faithful induction, indicating that their intelligence is mostly based on statistical correlations rather than causal understanding.

In analogical tasks such as Bongard Problems, LVLMs over-rely on local features (color, quantity) and ignore high-level relational structures; even when perception is successful, they struggle to maintain relational alignment, minor changes lead to performance degradation, and they often substitute literal descriptions for true relational transfer, showing "pseudo-understanding".

LVLMs face difficulties in algorithmic reasoning (multi-step planning) and deductive reasoning (logical deduction): they struggle to maintain long-range logical consistency, multi-step reasoning easily accumulates errors; spatial reasoning is limited by the granularity of visual encoders, affecting practical applications such as physical scene understanding.

The analysis found that LVLMs have common reasoning problems: sensitivity to distribution shifts (overfitting to training statistics), entanglement of perceptual bottlenecks and reasoning defects, and a disconnect between language fluency and reasoning fidelity (hallucinatory explanations). These deep-seated issues restrict true reasoning capabilities.

The researchers propose improvement directions: developing perception-reasoning decoupling methods, building training data covering diverse distributions, exploring architectural innovations such as neuro-symbolic approaches, and evolving evaluation protocols to capture deep reasoning dimensions, promoting AI to move from pattern matching to true understanding.