A New Framework for Video Understanding with Multimodal Large Language Models: The Trinity of Watching, Memory, and Reasoning

This article introduces a new MLLM video understanding framework from a human perspective, with three core capabilities: "watching", "memory", and "reasoning". The original authors are arXiv authors, source platform is arXiv, original title is Watch, Remember, Reason: Human-View Video Understanding with MLLMs, link: http://arxiv.org/abs/2606.07433v1, release time: 2026-06-05T16:29:13Z. This framework systematically sorts out the technical challenges and solutions of current video multimodal large models in spatiotemporal perception, long video processing, memory modeling, and faithful reasoning.

Traditional video analysis methods often split tasks into independent benchmark tests, while MLLM methods understand video content from a macro perspective. As research expands to long videos, multimodal, and knowledge-intensive scenarios, models need to address challenges such as sparse evidence, long-range dependencies, multimodal alignment, and reliable reasoning under limited computation. The framework proposed in this article decomposes video understanding into three core capabilities—watching, memory, and reasoning—providing a unified analytical structure and systematic methodology.

"Watching" is the foundation of video understanding, covering the ability to extract perceptual representations from raw videos:

1. Fine-grained spatiotemporal perception: Capture spatial details (object position/appearance) and temporal dynamics (actions/changes) using strategies like Transformer spatiotemporal attention, 3D convolution, and video encoders.
2. Efficient processing: For long videos, balance quality and computational cost through sparse sampling of key frames, hierarchical processing, and progressive encoding.
3. Audio-visual joint perception: Use early/mid/late fusion strategies to integrate visual and auditory cues for complete scene understanding.

"Memory" addresses the context preservation problem for long videos:

1. Offline memory: For complete videos, design compact memory vectors (key frames/event segments/implicit representations) and structured storage strategies for efficient retrieval.
2. Streaming memory: In real-time scenarios, achieve incremental updates and historical references through sliding windows, memory compression, and selective forgetting.
3. Long-range dependency modeling: Use approximate attention, hierarchical attention, and external memory expansion to solve the computation/memory bottlenecks of Transformers in ultra-long videos.

"Reasoning" transforms perception and memory into meaningful outputs:

1. Text reasoning: Perform temporal (event sequence), causal (event relationship), and logical (multi-step inference) reasoning based on video features.
2. Video-assisted reasoning: Dynamically review video clips to retrieve information, simulating the human cognitive process of "thinking while watching".
3. Faithfulness and interpretability: Ensure conclusions are supported by videos through attention visualization, evidence chain tracking, and explicit evidence citation to enhance transparency.

Application domains of video MLLMs include:

• First-person perspective videos: Life assistance, health monitoring;
• Sports event analysis: Tactical analysis, highlight extraction, commentary generation;
• Educational video understanding: Intelligent Q&A, knowledge point extraction, learning path recommendation;
• Medical video analysis: Surgical video processing, auxiliary diagnosis and education;
• Narrative video understanding: Content recommendation, plot analysis, summary generation. Evaluation benchmarks cover dimensions such as various task types (from action recognition to open Q&A), video lengths (from short to several hours), and modal combinations (single/multimodal).

Current challenges in the field:

1. Scalability: Computation/memory bottlenecks when processing hour-long videos;
2. Memory-perception architecture: More efficient explicit/implicit memory mechanisms;
3. Evidence-anchored reasoning: Ensure reasoning is anchored to video evidence to avoid hallucinations;
4. Cross-modal alignment: Better alignment of visual, auditory, and language modalities;
5. Real-time interaction: Support streaming input and real-time responses. Conclusion: This framework provides a clear roadmap for video MLLMs. Enhancing the three core capabilities is expected to build human-level understanding systems. For related resources, please follow https://github.com/marinero4972/Awesome-HumanView-VideoUnderstanding.