CLVG-Bench: A Systematic Evaluation Framework for Multimodal Reasoning Capabilities of Video Models

CLVG-Bench is a systematic evaluation framework targeting the gap in multimodal reasoning capabilities of current video generation models. It introduces a new evaluation paradigm for context learning-based video generation, and reveals the real limitations of SOTA video models (such as Sora, Runway Gen-3, etc.) in physical reasoning, causal reasoning, and other aspects through an adaptive video evaluator, promoting the shift of video generation evaluation from "quality-oriented" to "capability-oriented."

Current video model evaluation mainly focuses on visual quality (e.g., FID, FVD) and human preference scores, but fails to test the model's true understanding of logical relationships, physical laws, and causal reasoning in text instructions. For example, a model may generate a visually coherent video that violates physical laws (such as a ball accelerating uphill). The CLVG-Bench team proposes the "Context Learning Video Generation (CLVG)" paradigm, aiming to evaluate the model's ability to simulate and reason about real-world dynamics.

1. Context Learning Video Generation: Breaks the traditional "text→video" mapping, requiring the model to infer subsequent developments based on contextual examples, which is closer to human learning methods and tests internal understanding rather than superficial imitation.
2. Adaptive Video Evaluator: Based on a small amount of manual annotations, dynamically adjusts evaluation strategies, balances the accuracy of human judgment and the scalability of automatic evaluation, and solves the problem of open-domain video evaluation.

CLVG-Bench covers five major evaluation dimensions:

• Spatial Reasoning: Object position, movement direction, spatial relationships (e.g., an object moving from left to right and away from the camera);
• Temporal Reasoning: Event sequence, duration, speed changes (e.g., movement that starts slow then becomes fast);
• Physical Reasoning: Laws such as gravity, friction, collision (e.g., parabolic trajectory of a projectile);
• Causal Reasoning: Causal relationships between events (e.g., rain causing the ground to get wet);
• Compositional Reasoning: Comprehensive ability across multiple dimensions (e.g., complex scenes combining spatial, physical, and causal aspects). Test cases from simple to complex are designed for each dimension.

Through CLVG-Bench evaluation, it is found that SOTA models have significant limitations:

1. Insufficient understanding of physical laws: Difficulty in accurately simulating motion trajectories, collisions, gravity, etc., with performance on related tasks lower than human levels;
2. Weak causal reasoning ability: Only captures the temporal sequence of events, unable to establish true causal connections;
3. Lack of long-range consistency: When generating long videos or multi-step reasoning videos, the probability of logical contradictions increases significantly with length. These indicate that models rely more on statistical patterns in training data rather than understanding the laws of the world.

1. Simply expanding model size and data volume cannot solve reasoning defects; structured training data with causal/physical annotations is needed;
2. Video understanding and generation should be deeply integrated to achieve true multimodal reasoning capabilities;
3. The evaluation system needs to evolve in sync with capability development, and CLVG-Bench provides a rigorous direction for the field.

Currently, the CLVG-Bench code and dataset are being prepared for release, and the complete evaluation code and benchmark dataset will be open-sourced. In the long run, CLVG-Bench promotes the shift of video generation evaluation from "quality-oriented" to "capability-oriented," providing a basic tool for evaluating the reasoning capabilities of video models in fields such as entertainment, education, and simulation.