PAI-Bench 2: A New Paradigm for Evaluating Video Generation Models' Physical World Understanding

PAI-Bench 2 is the first comprehensive benchmark focusing on evaluating video generation models' physical world understanding ability. It shifts the evaluation paradigm from surface visual quality to physical correctness, using a HybridJudge architecture that combines an analytical validator (PhysicsJudge) and multi-LLM ensemble judge to assess whether generated videos conform to real physical laws across five tracks.

Recent video generation models like Sora excel in visual quality but often lack true physical world understanding. PAI-Bench v1 had key limitations: it used a single MLLM (Qwen3-VL) as judge, leading to black-box results, limited physical knowledge scope, and no way to quantify result credibility.

PAI-Bench 2 introduces the HybridJudge architecture: 1. PhysicsJudge (analytical validator) handles rigid body, contact, fluid dynamics with specific checks (e.g., gravity alignment for rigid bodies, mass conservation for fluids); 2. EnsembleJudge (multi-LLM integration) is used when scenes are unparseable, with median/mean voting, consistency metrics, and disagreement marking.

The scoring formula is G_score = 0.3×Quality_Score +0.7×Domain_Score. Quality_Score covers 6 dimensions (subject consistency, motion smoothness, etc.). Domain_Score (physical correctness) uses supplementary metrics like flow smoothness (Farneback flow), depth stability (DepthAnythingV2), and posture validity (MediaPipe).

The annotation process uses human-machine collaboration (MLLM drafting + ≥3 human reviews with ≥0.8 agreement). Five tracks: Track G (unconditional generation), Track C (conditional generation), Track U (video understanding), Track CF (counterfactual reasoning), Track DV (dynamic vision).

Installation steps: git clone repo → create venv → install dependencies (including PyAV for MLLM calls). Model interface requires a Python function returning video path. Run commands: pai-bench run (generate results) and pai-bench score (compute scores).

Known limitations: threshold calibration (based on synthetic data), ViCLIP weight issue (penalizes real footage), lack of time granularity. Future directions: marginal case cross-validation, per-2-second window scoring, time labeling of physical phenomena.

Academic value: standardized framework, interpretable metrics, error analysis tools. Industry value: model selection, training feedback, safety assessment. Conclusion: PAI-Bench 2 drives the shift from visual mimicry to physical understanding, catalyzing robust video generation AI progress.