Q-Fold: Query-Aware Focus-Context Spatiotemporal Folding for Long Video Understanding

Q-Fold is a training-free input construction framework for long video understanding. Through query-aware heterogeneous focus-context representation, it simultaneously preserves high-fidelity visual evidence and broad temporal coverage under a limited visual budget, achieving a performance improvement of up to 9.1 percentage points on ultra-long video benchmarks. This framework breaks the traditional frame-centric paradigm, uses continuous time segments as units, and can be combined with existing Video-MLLMs without additional training costs.

Long video understanding is a core challenge for Video-MLLMs. Time-extended videos contain thousands of frames, and exhaustive processing is computationally unaffordable. Existing methods mostly follow the frame-centric paradigm but use similar representations for retained content, failing to balance high-fidelity visual evidence and broad temporal coverage—leading to either loss of key details or omission of important temporal context.

Q-Fold uses continuous time segments as basic units and constructs heterogeneous representations under query guidance: 1. Focus frames: Retain high-fidelity frames for segments highly relevant to the query to ensure no loss of key visual evidence; 2. Context layout: Fold low-relevance segments into compact representations that maintain temporal order, preserving broad temporal coverage. This method balances key details and temporal context while maintaining local temporal continuity.

Key innovations of Q-Fold include: 1. Query-aware selection mechanism: Leverage the capabilities of existing multimodal large models to evaluate video segment relevance based on queries without additional training; 2. Spatiotemporal folding strategy: Compress low-relevance segments into context representations that maintain temporal order, reducing input volume while preserving temporal structure information.

In four long video benchmark tests, Q-Fold combined with various Video-MLLMs achieved performance improvements without increasing the input budget. Among them, on ultra-long video benchmarks, the performance improvement reached up to 9.1 percentage points. As a training-free framework, it can be combined with any existing Video-MLLM without additional training costs.

Technical significance: 1. Balance efficiency and effectiveness, achieving high efficiency and performance under limited budgets; 2. Strong versatility, adaptable to various Video-MLLMs without training; 3. Good interpretability, as the focus-context distinction makes the model's attention areas more transparent. Potential application scenarios: Long video content analysis and summarization, intelligent retrieval of surveillance videos, educational video understanding and Q&A, automatic commentary for sports events.

Q-Fold provides an efficient solution for long video understanding through query-aware heterogeneous representation methods. It breaks the frame-centric paradigm, uses continuous time segments as units, and preserves key information while achieving broad temporal coverage. This work not only brings significant performance improvements but also demonstrates a new idea of intelligent input construction strategies to unleash the potential of multimodal large models, which will play an important role in video understanding applications in the future.