A Study on the Systematic Auditing Framework for Physical Reasoning Capabilities of Vision-Language Models

Newcastle University developed a visual auditing system based on the violation-of-expectation framework. Using the classic Shell Game task, it tests the capabilities of cutting-edge Vision-Language Models (VLMs) in object permanence, temporal continuity, and hidden state reasoning, revealing key issues such as fundamental limitations in physical understanding and calibration gaps in current models.

Traditional AI safety frameworks mainly focus on static inputs or semantic-level digital perturbations. This study focuses on a deeper issue: whether the internal world model of AI agents is consistent with continuous physical reality. The motivation comes from observing that cutting-edge VLMs often make errors inconsistent with human intuition when handling scenarios involving object position tracking, occlusion understanding, and hidden state reasoning. The aim is to quantify these defects and explore their root causes through a standardized testing framework.

Core methodological innovations include: 1. A two-stage prediction framework (decoupling raw sensory observation from final reasoning, and forcing the recording of temporal tracking evidence to test object permanence); 2. A design for immediately verifiable outputs (continuously parsing text generation streams into structured data to support automated evaluation). A frugal AI approach is adopted, prioritizing testing of quantized open-source models with 4-12 billion parameters, running with mixed precision on edge devices. The technical implementation is a modular end-to-end pipeline: video processing (Decord hardware-accelerated frame sampling), spatial normalization (Pillow adaptive padding), and reasoning parsing (regular expressions + PyYAML to solve format anomaly issues).

Evaluation metrics include Expected Calibration Error (mapping the difference between subjective confidence and objective tracking accuracy), Prediction Entropy (the degree of hesitation in the model's internal probability distribution), and Empirical Label Entropy (output variance from repeated trials). Experimental results show: Gemini 3 Flash has poor reasoning accuracy but is overconfident (average confidence 92.19%); Pixtral 12B has the best balance of stability and certainty; Qwen3-VL-4B shows uncertain and unstable performance.

Key findings: Large proprietary flagship models exhibit severe structural overconfidence under occlusion conditions, and their confidence outputs are unreliable. Typical model biases include: late temporal decay (losing object positions in later tracking stages), center stage effect (focusing attention on the center of the frame), biomechanical distraction (being distracted by human body movements), and early grounding collapse (failing to correctly establish object-position correspondence initially).

Engineering contributions: Providing a complete open-source engineering pipeline (including analysis notebooks, experimental data, multi-model results, and automated evaluation scripts) to support reproduction and extension. Research significance: Offering important tools and methodologies for VLM safety assessment, and revealing the boundaries of physical reasoning capabilities. Future directions: Extending to more complex physical scenarios, testing models' understanding of concepts such as causality and conservation of quantity.