VSTAT: Diagnosis of Visual State Tracking Capabilities in Video Understanding for Multimodal Large Models

The VSTAT benchmark, released by the original author team (arXiv) on June 2, 2026, aims to diagnose the visual state tracking capabilities of multimodal large language models (MLLMs). The study found that although MLLMs excel at text reasoning, their visual perception capabilities are insufficient to effectively track changes in the state of entities in videos, and their performance on VSTAT is far below human levels. This benchmark fills a gap in the existing evaluation system and is of great significance for MLLM video understanding research and applications.

• Original paper link: http://arxiv.org/abs/2606.03920v1
• Release date: June 2, 2026

When humans watch videos, they continuously track the temporal changes of entities, states, and events (visual state tracking), which is the cognitive foundation of video understanding. However, existing MLLM evaluation systems ignore this capability, focusing more on single-frame recognition, action classification, or short-segment understanding, leading to the situation where the excellent performance of models on existing benchmarks does not reflect their real video understanding capabilities.

The VSTAT benchmark follows three core principles:

1. Dependence on continuous perception: Questions need to integrate information from the entire video stream, eliminating shortcuts using single frames or short segments;
2. Coverage of synthetic and real scenes: Includes 834 video clips (synthetic + real) to ensure comprehensive evaluation;
3. Focus on state change reasoning: 1500 questions involve core cognitive operations such as entity recognition, attribute changes, and causal relationships.

• Model performance: Human accuracy is nearly perfect, while the best MLLM only reaches a moderate level, far below humans;
• Reasons for failure: Correct text reasoning but failed visual perception (insufficient alignment between visual encoder and language module);
• Agent methods are ineffective: Tool calling/code generation cannot compensate for basic visual perception defects.

VSTAT is the first to systematically evaluate visual state tracking capabilities, revealing the core flaw of MLLMs: the lack of true visual state tracking capabilities, rooted in insufficient visual-language alignment rather than reasoning ability issues. This finding has safety implications for the deployment of MLLM applications (such as autonomous driving and surveillance).

Technical directions:

1. Improve the temporal modeling capability of visual encoders;
2. Enhance visual-language alignment mechanisms;
3. Develop specialized video reasoning architectures;
4. Extend VSTAT to more complex tasks.

Application recommendations: In scenarios requiring state tracking such as autonomous driving and medical imaging, multimodal fusion strategies should be adopted, and traditional CV methods should be combined to verify model outputs.