SlotVTG: Object-oriented Video Temporal Grounding Adapter with Significant Cross-domain Generalization Improvement

The SlotVTG framework addresses the cross-domain generalization challenge of multimodal large language models (MLLMs) in video temporal grounding (VTG) tasks using a lightweight object-centric adapter. This method guides MLLMs to perform object-level visual reasoning without retraining the entire model, significantly improving generalization on out-of-domain data.

Video Temporal Grounding (VTG) is a core task in multimodal understanding, requiring the localization of event time boundaries in videos based on natural language descriptions. While MLLMs perform well in this task, they face the issue that coarse-grained recognition cannot support fine-grained temporal understanding. Traditional task-specific fine-tuning tends to make models memorize dataset shortcuts, leading to extremely poor generalization on out-of-domain (OOD) data—for example, significant performance drops across datasets.

Object-centric learning decomposes scenes into entity-level representations, allowing models to focus on specific objects and their interactions instead of relying on statistical correlations for prediction, providing a direction to solve cross-domain generalization. However, existing object-centric methods require running multi-stage training pipelines from scratch, which incurs high computational and time costs, limiting their practical application and popularization.

Core Technical Mechanisms

1. Slot Decomposition: Decompose visual tokens into abstract slots via slot attention mechanism, where each slot represents a potential object or concept.
2. Sequence Reconstruction & Object Priors: Reconstruct the original visual sequence using decomposed slots, and introduce objectness priors from self-supervised visual models to encourage slots to form semantically coherent clusters (corresponding to real physical objects).

Architectural Advantages

• Plug-and-play: Directly insert into pre-trained MLLMs without modifying original weights
• Computationally efficient: Training cost is much lower than retraining multi-stage pipelines
• High interpretability: Slot representations intuitively reflect the objects the model focuses on

Cross-domain evaluation results from the research team on standard VTG benchmark datasets show:

1. Improved cross-domain generalization: Models equipped with SlotVTG are more robust and accurate in localization when facing out-of-domain test sets.
2. Preserved in-domain performance: While improving generalization ability, in-domain performance is comparable to the original model.
3. Low overhead: The introduced computational overhead is minimal, suitable for resource-constrained scenarios.

The technical significance and application prospects of SlotVTG include:

1. Lowering the adoption threshold of object-centric methods and accelerating related research progress.
2. Enhancing the reliability of MLLMs in real scenarios and reducing the demand for domain-specific labeled data.
3. The design concept can be extended to other multimodal tasks such as visual question answering and video captioning.

SlotVTG still has directions to explore:

1. Adaptive selection of slot count: Dynamically adjust the number of slots based on video complexity.
2. Integration of richer prior knowledge: Introduce priors from dimensions such as actions and scenes.
3. Optimization for long video processing: Efficiently handle long videos with numerous objects and complex temporal structures.