Thinking in Dynamics: How Multimodal Large Language Models Perceive, Track, and Reason About Dynamic Changes in the 4D Physical World

This article introduces a groundbreaking study accepted by CVPR 2026, which proposes the Dyn-Bench benchmark to systematically evaluate, for the first time, the ability of multimodal large language models (MLLMs) to perceive, track, and reason about spatiotemporal dynamics in the 4D physical world. It reveals key limitations of current models in dynamic scene understanding and directions for improvement.

Humans live in a 4D physical world and can understand the movement trajectories of objects, their interactions, and camera movements in dynamic scenes. Current MLLMs perform well in static visual understanding, but whether they excel at 'dynamic thinking' remains unclear—this is crucial for building embodied agents, autonomous driving systems, and robotic systems.

Dyn-Bench is a large-scale benchmark for evaluating MLLMs' dynamic understanding ability, containing 1000 videos (real + synthetic), 7000 visual question-answer (VQA) pairs, and 3000 dynamic object localization pairs. It evaluates from three key dimensions:

1. Camera-Object Dimension: Understand the movement of objects relative to the camera;
2. Inter-Object Dimension: Reason about object interactions and relative dynamics;
3. Object-Scene Dimension: Analyze object-scene interactions and evolution.

Evaluations of models like GPT-4V, Gemini, and Claude 3 reveal:

1. It's hard to balance language reasoning and visual localization;
2. There are contradictions in explaining motion interactions in complex scenes;
3. Traditional prompting strategies (e.g., Chain of Thought) have limited improvement effects.

Promising improvement directions include:

1. Mask-Guided Fusion: Incorporate object segmentation masks into reasoning to enhance dynamic object tracking ability;
2. Spatiotemporal Textual Cognitive Map (ST-TCM): Construct structured spatiotemporal relationship representations to simulate human spatiotemporal reasoning processes.

Research Significance:

• Embodied Intelligence: Provides tools for evaluating and improving perceptual foundations;
• Autonomous Driving: Offers references for the design of perception systems. Open-Source Contributions: HuggingFace dataset kairunwen/DynamicVerse, evaluation code, a framework supporting over 20 MLLMs, and an experimental leaderboard.

Evaluation Metrics:

• QA Accuracy: Measures the matching degree of answers in VQA tasks;
• Mask J&F Score: Combines IoU and boundary F-measure to evaluate localization accuracy. Supported Models: Covers over 20 mainstream MLLMs such as Sa2VA series, InternVL3/3.5, Qwen2.5-VL, LLaVA-OneVision, etc.