AOT: Efficient Token Compression for Video Large Models via Local and Global Context Optimization

AOT is a CVPR 2026 work proposed by Adobe Research. Its core lies in jointly optimizing local and global visual contexts to significantly reduce the number of tokens in video large language models while preserving understanding capabilities, thus improving inference efficiency. This article will analyze it from dimensions such as background, methods, implementation, experiments, and applications.

Computational Bottlenecks in Video Understanding

Video Large Language Models (Video LLMs) are widely used in scenarios such as subtitle generation and visual question answering. However, the temporal dimension of videos leads to an explosion in the number of tokens (short clips can reach tens of thousands of tokens), increasing computational costs and limiting real-time applications. Existing token compression methods face a dilemma: over-compression loses key information, while insufficient compression fails to solve the computational bottleneck. Balancing compression ratio and understanding capability is the core challenge.

Core Innovations of AOT

The innovation of AOT (Adaptive Optimal Tokenization) lies in the joint optimization of local and global contexts:

• Local Optimization: For single frames/short time periods, identify key regions and adaptively allocate token budgets—retain details in information-rich regions and aggressively compress redundant regions;
• Global Optimization: Identify key frames and temporal nodes across the time dimension, avoid uniform allocation of computational resources, and prioritize token budgets for key segments. This strategy achieves a significant reduction in tokens while maintaining or even improving understanding capabilities.

Technical Implementation and Architecture Design

AOT is based on the LLaVA-NeXT architecture, with core modules including:

• LLaVA-NeXT Module: Provides video-language alignment and dialogue interfaces;
• visionzip Module: Implements token compression algorithms for local/global context analysis;
• lmms_eval Module: Integrates a standardized evaluation framework;
• scripts Module: Training/inference startup scripts. The project includes training logs and visualization resources to facilitate reproduction and understanding.

Experimental Validation and Performance

AOT is evaluated on typical task benchmarks such as video question answering, subtitle generation, and temporal localization (specific data to be announced). The model weights use the Adobe Research License, and the code uses the MIT License. This dual licensing model balances research openness and commercial flexibility.

Application Scenarios and Practical Value

The value of AOT is reflected in:

• Long Video Platforms: Reduce inference costs in scenarios like online education and sports analysis, supporting real-time analysis;
• Edge Devices: Adapt to memory/computational constraints for efficient deployment;
• Technical Reference: The local-global joint optimization approach can provide references for efficiency optimization of multimodal models.

Project Status and Usage Recommendations

The AOT project is currently in the "cleanup and organization" phase, with code and documentation still being optimized. Recommendations:

• Follow subsequent updates for a stable experience;
• Refer to the arXiv paper (arXiv:2603.01400) and the project homepage for in-depth understanding;
• Developers familiar with the LLaVA-NeXT/lmms-eval frameworks can get started quickly.

Conclusion

AOT represents an important progress in efficiency optimization of video large models, balancing compression ratio and understanding capability through a local-global joint strategy. As the proportion of video content rises, such efficient technologies will play a key role and are worth the attention of researchers and engineers.