SVBench: An Evaluation Benchmark for Social Reasoning Capabilities of Video Generation Models

SVBench is a CVPR 2026 paper project and the first evaluation benchmark specifically designed for the social reasoning capabilities of video generation models, filling the gap in assessment standards for this field. It focuses on models' understanding of social common sense and norms, pushing video generation from visual realism towards compliance with social logic. The original project is maintained by Gloria2tt and published on GitHub (link: https://github.com/Gloria2tt/SVBench-Evaluation) on June 3, 2026.

In recent years, video generation technology has boomed (e.g., Sora, Runway Gen-3), but existing evaluations only focus on visual quality (FID/FVD) and text alignment, ignoring social common sense (such as rule violations like back-to-back conversations or making noise in libraries). Social reasoning capability is key to the practical application of models, hence the birth of SVBench.

SVBench evaluates from 5 key dimensions: spatial relationship understanding (e.g., conversation positions), behavioral normativity (scene-appropriate behaviors), role consistency (identity-matching behaviors), emotional expression rationality (consistency between facial expressions/body language and scenes), and social interaction logic (eye contact/turn-taking in conversations, etc.). The dataset covers daily scenarios (family dinners) to specific ones (courtrooms), with each use case containing clear social expectations.

SVBench adopts a hybrid evaluation approach: some aspects like spatial relationships can be automatically detected (e.g., character orientation/distance); others like emotional expression and interaction fluency require standardized human scoring. Meanwhile, fairness is ensured through unified prompts and comparative evaluations, and fine-grained error analysis (e.g., spatial errors, role inconsistency) is provided to help improve models.

Based on SVBench evaluations: 1. State-of-the-art models still make common sense mistakes in complex social scenarios; 2. Different models show varying performance across dimensions (e.g., good at spatial relationships but weak at behavioral norms); 3. There is a non-linear relationship between model size and social reasoning—targeted optimization is needed instead of just scaling up.

For research: It clarifies the optimization goals for social reasoning; For applications: It provides developers with a basis for model selection (e.g., virtual customer service needs behavioral norms, advertisements need spatial relationships); For ethics: It helps assess the risk of inappropriate content and contributes to AI safety.

Current limitations: It covers more static scenarios and fewer dynamic interactions (continuous conversations/conflicts); cultural specificity (mainly Western scenarios). Future directions: Expand dynamic scenarios and cross-cultural datasets, develop more automated evaluation methods, and integrate social reasoning into model training objectives.