BilliardPhys-Bench: A New Benchmark for Testing Physical Reasoning Capabilities of Multimodal Large Models

The research team has launched the BilliardPhys-Bench benchmark for billiards physical reasoning, evaluating the ability of multimodal models such as GPT, Claude, Gemini, and Qwen to predict object motion and collision reasoning, and identifying systematic flaws like the "static bias". Physical reasoning is key for AI to achieve true intelligence and apply to scenarios like robotics and autonomous driving, and this benchmark provides a strictly controllable platform for related evaluations.

Current multimodal large language models (MLLMs) perform well in static image recognition, but their intuitive physical reasoning ability is weak. Predicting object motion and interactions from a single image remains a major challenge. Physical reasoning is the core of human cognition and is crucial for AI's intelligent upgrade and applications like robotics and autonomous driving.

Benchmark Introduction

The billiards scene is chosen (with clear physical laws, controllable variables, and verifiable results). A procedural engine generates random scenes with real physical parameters, and the model can be evaluated by adjusting complexity, simulation duration, and geometric structure.

Three Testing Dimensions

1. Ball-Ball Collision Prediction: Infer the direction of velocity, predict the trajectory after collision, testing conservation of momentum and spatial reasoning;
2. Wall Rebound Reasoning: Predict the reflection direction of the ball after hitting the table edge, testing the application of the law of reflection;
3. Final Position Estimation: Predict the stopping position of the ball by integrating multiple factors, testing long-range physical reasoning.

Performance Declines with Complexity

All mainstream models (GPT, Claude, Gemini, Qwen series) have significant limitations in physical reasoning performance. Their accuracy drops sharply as scene complexity increases, relying on superficial statistical correlations rather than causal reasoning.

Static Bias Phenomenon

Models tend to predict "no interaction" (balls remain stationary or move in the original direction). This stems from the high proportion of static images in training data, leading to a default "no change" strategy, which does not meet the needs of physical reasoning.

Current models lack built-in physical inductive biases (such as priors like object continuity and conservation of momentum), relying on data to learn physical laws rather than built-in constraints. The Transformer architecture is designed for general sequence modeling; it is flexible but needs to learn physical laws from scratch, lacking physical intuition.

1. Introduce Physical Priors: Embed physical constraints into the model (e.g., enforce conservation of momentum in the network structure);
2. World Model Learning: Train explicit physical prediction modules to model object dynamics;
3. Enhance Causal Reasoning: Introduce causal mechanisms to understand the causal relationships of physical events;
4. Optimize Dynamic Attention: Improve the attention mechanism to model continuous motion and interactions of objects.

BilliardPhys-Bench provides a strict platform for evaluating the physical reasoning of multimodal AI, revealing the gaps in existing models. As AI applications in the physical world increase, improving physical reasoning capabilities will become a core research focus. This benchmark and subsequent work will provide guidance for the progress of the field.