Video-Zero: A New Video Understanding Method Based on Temporal Evidence Self-Evolution

Video-Zero is an annotation-free question-answering co-evolution framework. Its core lies in the Questioner identifying information-rich temporal evidence segments and generating questions that depend on these segments, while the Solver learns to answer and align with the evidence. This method consistently improves the performance of multiple Video Large Language Model (Video VLM) backbones across 13 video understanding benchmarks, providing a new path for the video understanding field to break free from reliance on manual annotations.

Video understanding requires processing temporal dimension information (action evolution, event causality, etc.), but existing Video VLMs heavily rely on expensive manually annotated data. The self-evolution paradigm has shown potential in the text domain, but extending it to video faces three major challenges: video length redundancy, temporal sparsity (small proportion of key evidence), and dynamic changes. Moreover, simply transferring text self-evolution methods leads to supervision signals lacking temporal grounding, which fails to truly enhance temporal reasoning capabilities.

Video-Zero adopts a dual-component collaborative design:

• Questioner: Analyzes videos to identify information-rich evidence segments (based on visual saliency, semantic importance, and temporal distribution), and generates questions that must rely on these segments (e.g., "Did the person drink water before or after picking up the cup?");
• Solver: Answers questions and locates evidence, with training objectives including answer correctness and evidence alignment;
• Collaborative Cycle: Initialization → Evidence Discovery → Question Generation → Answer Verification → Feedback Update → Iteration. Bidirectional feedback enhances the capabilities of both components.

Key technologies of Video-Zero include:

1. Hierarchical Temporal Evidence Representation: Segment-level (coarse-grained event regions), frame-level (fine-grained localization), and cross-frame relationships (capturing action evolution);
2. Evidence-Aware Attention Mechanism: Dynamically focuses on video segments relevant to the question, improving reasoning efficiency;
3. Progressive Difficulty Curriculum: From simple temporal localization to complex reasoning, ensuring stable training and mastery of basic capabilities.

Excellent performance across 13 benchmarks:

• Temporal Localization: ActivityNet Captions accuracy increased by 15-20%, and Charades-STA more accurately locates action boundaries;
• Long Video Understanding: MovieNet/YouCook2 QA accuracy increased by over 25%, effectively filtering redundancy;
• Video Reasoning: NEXT-QA/Causal-VidQA performance is comparable to supervised learning, with significant improvements in causal reasoning;
• Cross-Model Transfer: Consistently improves the performance of backbones like CLIP, VideoMAE, and InternVid, verifying the value of the paradigm.

Current limitations: High computational cost (large overhead in the iteration process), lack of automatic evidence quality evaluation metrics, no integration of multimodal information (audio/subtitles), and unvalidated open-domain generalization. Future directions: Optimize computational efficiency, develop evidence quality evaluation mechanisms, expand multimodal fusion, and validate open-domain generalization capabilities.

Core insights from Video-Zero: In temporal tasks, the grounding of supervision signals is more important than difficulty; co-evolution breaks through the limitations of single components; it proves the feasibility of high-quality unsupervised learning in the video domain. This framework provides a feasible path for video understanding to break free from reliance on manual annotations, offers new ideas for self-supervised learning research, and helps build more powerful video AI systems.