ChronoPhyBench: Do Multimodal Large Models Truly Understand the Physical World, or Are They Just Leveraging Linguistic Priors?

ChronoPhyBench is a brand-new multimodal physical dynamic reasoning benchmark designed to test whether Multimodal Large Models (MLLMs) truly possess cross-modal physical reasoning capabilities or merely rely on linguistic priors for "hallucinatory" reasoning. This benchmark effectively distinguishes between a model's real physical understanding and its reliance on linguistic shortcuts through sequential physical state prediction tasks. Experiments find that the physical reasoning capabilities of current open-source MLLMs are still in the initial stage, which has important guiding significance for the development of Physical AI and Artificial General Intelligence (AGI).

Source: arXiv 2026-06-06, Link: http://arxiv.org/abs/2606.07962v1

In recent years, MLLMs have performed prominently in open-world reasoning and multimodal tasks (such as visual question answering and image captioning), but core issues remain unresolved: Do models truly integrate cross-modal information to build physical reasoning chains, or do they only use linguistic priors to mask unimodal dependencies? If relying solely on linguistic priors, models will be limited in scenarios requiring precise physical reasoning, such as robot control and physical simulation. Existing benchmarks cannot effectively distinguish between cross-modal reasoning and linguistic shortcuts, leading to evaluation results that fail to reflect the true boundaries of capabilities.

The core design of ChronoPhyBench combines next-state prediction with Visual Question Answering (VQA) to force models to perform cross-modal reasoning. It includes two tasks:

1. Single-frame Selection Task: Choose the next state that conforms to physical laws from candidate frames, testing understanding of laws such as object motion and collision;
2. Multi-frame Sequential Sorting Task: Arrange video frames in physical chronological order, testing the ability to model dynamic evolution.

Dataset scale: 10,000+ long video clips, 5 million tokens, covering various physical scenarios such as rigid body motion and fluid dynamics. Manual verification ensures physical correctness and annotation accuracy.

Experimental results show that current open-source MLLMs perform far below expectations on ChronoPhyBench, even models that excel in traditional VQA struggle. Error patterns are systematic:

• Tend to predict based on object appearance rather than physical laws;
• Generate inferences that violate physical common sense in complex dynamic scenarios. This indicates that existing models may rely heavily on linguistic priors rather than true physical understanding.

ChronoPhyBench has far-reaching implications for Physical AI:

1. Provides a robust and transparent evaluation framework to accurately measure physical reasoning capabilities;
2. Quantifies model hallucination rates, providing a basis for reliability assessment in physical interaction scenarios such as autonomous driving and robot operation;
3. Offers a new perspective for AGI research—true AGI needs to deeply understand the physical world, not just linguistic pattern matching.

Future research directions:

1. Improve Model Architecture: Explore architectures that integrate spatiotemporal information and physical constraints, rather than simply concatenating visual encoders and language models;
2. Introduce Physical Priors: Explicitly add physical law constraints during training to establish physical intuition representations;
3. New Training Strategies: Design dedicated training objectives and curriculum learning for physical reasoning;
4. Expand Evaluation Dimensions: Cover more physical fields such as quantum mechanics and relativity to comprehensively test capabilities.