MemDreamer: Using Hierarchical Graph Memory and Intelligent Retrieval Mechanism to Solve the Challenge of Long Video Understanding

MemDreamer is an innovative system that addresses the challenges of long video understanding by decoupling perception and reasoning. Its core uses a hierarchical graph memory architecture and an intelligent retrieval mechanism, achieving SOTA performance while using only 2% of the context, narrowing the gap with human experts to just 3.7 points and opening a new path for long video understanding.

Long video understanding is the ultimate challenge in AI vision: high-density information in videos leads to context explosion, unbearable computational costs, and failure of attention mechanisms due to overload (attention dilution). Existing solutions either aggressively compress to sacrifice details or process in segments which makes it hard to establish cross-segment connections—neither meets the requirements.

The core of MemDreamer lies in decoupling perception and reasoning:

1. Hierarchical Graph Memory: A three-layer architecture (perception layer extracts visual features, event layer organizes discrete events, relation layer captures event correlations), where the graph structure naturally expresses complex relationships;
2. Intelligent Retrieval Mechanism: Goal-oriented exploration, gradually locating relevant information through tool calls (node search, edge traversal, etc.), similar to human recall thinking.

Processing is divided into two stages:

• Memory Construction: Incrementally process video segments, extract 2D+3D features using the UniK3D architecture, organize them into Gaussian primitives (continuous probability representation), and continuously expand and fuse memory;
• Query Reasoning: Adopt an observation-reasoning-action loop, form hypotheses based on the problem, call retrieval tools to obtain information, iteratively update understanding until an answer is reached, supporting multi-hop queries.

MemDreamer achieved SOTA on four mainstream long video understanding benchmarks, using only 2% of the context (e.g., processing only 1.2 minutes of a 1-hour video), narrowing the gap with human experts to 3.7 points. Ablation experiments verified the synergistic advantages of the hierarchical graph structure and intelligent retrieval, and it has good cross-benchmark generalization ability.

Statistical analysis shows that there is a strong positive linear correlation between performance on logical reasoning benchmarks and long video understanding performance. This indicates that long video understanding is not only a memory problem but also a complex logical reasoning problem (causal relationships, implicit information, multi-step reasoning), and MemDreamer's success stems from transforming it into a reasoning problem.

MemDreamer can be applied to:

• Video content analysis: Efficient media asset management, natural language search for videos;
• Surveillance security: Analyze long surveillance videos and identify abnormal events;
• Education: Automatically analyze teaching videos, generate summaries, and answer questions;
• Entertainment: Support interactive video experiences and explore plot clues.

Limitations: High computational overhead in the preprocessing stage, need for improvement in fine visual detail queries, and evaluation focused on question-answering tasks. Future Directions: Optimize memory construction efficiency, explore lightweight graph representations, expand video types (360-degree, multi-view), real-time video processing, and enhance reasoning by combining with external knowledge bases.