StreamDyCoke: Breakthrough in Dynamic Token Compression for Video Large Language Models

StreamDyCoke is the streaming extension of DyCoke, a CVPR 2025 paper, and is a dynamic token compression technology designed for real-time video large language models. Through an on-demand compression strategy, it significantly reduces computational overhead while maintaining model performance, solving the token explosion problem caused by the high dimensionality of video data and meeting the needs of real-time applications.

With the development of Multimodal Large Language Models (MLLMs), video understanding has become a frontier, but the high-dimensional nature of video brings computational challenges: a few seconds of video contains hundreds of frames, and independent input of each frame leads to token count swelling to thousands or even tens of thousands. This causes increased inference latency (unable to achieve real-time performance) and a sharp increase in computational resource consumption (limiting edge deployment). Traditional uniform sampling or fixed frame dropping methods reduce tokens but easily lose key information, leading to performance degradation.

The core of DyCoke (Dynamic Compression) is "on-demand compression": dynamically adjusting the token density of each frame according to content complexity. A lightweight policy network is introduced to evaluate the visual information amount of each frame and determine the number of tokens to retain. Tokens are significantly compressed in static/slow-changing scenes, while more details are retained in frames with intense motion or rich information. This mechanism reduces the average token count by more than 50% while maintaining high accuracy.

StreamDyCoke is the streaming extension of DyCoke, optimized for real-time video streams to address the limitation of the original version requiring global analysis. Key improvements include three aspects:

1. Sliding window policy network: Maintains a fixed-size historical frame buffer; the policy network makes decisions based on a local window, reducing time complexity from O(N²) to O(W²) (W is much smaller than the total number of frames);
2. Online token caching mechanism: Caches the compressed representation of processed frames; new frames only compute differential tokens to reduce redundant calculations;
3. Adaptive frame rate adjustment: Spatiotemporal joint optimization—reduces sampling frequency during stable content periods and increases it when changes are intense.

StreamDyCoke is based on the PyTorch framework, compatible with mainstream video LLMs (such as LLaVA-Video, Video-LLaMA), and provides a plug-and-play compression module. Evaluations show: when the average token count is reduced by 60%, the accuracy drops by less than 2%; end-to-end latency is reduced from hundreds of milliseconds to tens of milliseconds, meeting the requirements of 30fps real-time processing. Moreover, the compression strategy is learnable—compression patterns can be optimized for specific tasks (action recognition, video question answering, etc.) through end-to-end training.

Application scenarios: Video surveillance (real-time anomaly detection and early warning), autonomous driving (low-latency safety decision-making), mobile devices (reducing power consumption and bandwidth). Future directions: Co-design with hardware acceleration, joint compression of multimodal tokens (video/audio/text), adaptive compression strategy learning for specific domains. Efficient token compression will become a key component of multimodal AI infrastructure.