A Survey of Token Compression Techniques for Multimodal Large Language Models: Cutting-Edge Exploration Toward Efficient MLLMs

This article provides a survey of token compression techniques for multimodal large language models (MLLMs), aiming to analyze how to improve model efficiency while maintaining performance by reducing the number of visual tokens. It discusses the necessity of token compression, core challenges, mainstream technical routes, practical application prospects, and future development directions, providing references for the research and deployment of efficient MLLMs.

Necessity

With the rapid development of MLLMs, the large number of visual tokens generated by high-resolution image processing leads to huge computational overhead, limiting the model's ability to handle long sequences. Token compression has become a key direction to address this bottleneck.

Core Challenges

1. Visual information has high spatial redundancy;
2. Need to balance compression ratio and preservation of fine-grained details: excessive compression easily loses key features, while insufficient compression fails to leverage efficiency advantages.

Current mainstream technical routes include:

Sampling-based Sparsification Methods

Identify and retain the subset of tokens with the richest information, dynamically selected via attention mechanisms or importance scoring (e.g., prioritizing foreground objects).

Aggregation-based Token Merging Strategies

Aggregate semantically similar/spatially adjacent tokens into a single representative token, preserving the overall information of the merged region (soft merging/hard merging).

Knowledge Distillation and Lightweight Visual Encoders

Design efficient encoders that learn the capabilities of large encoders via knowledge distillation, output fewer feature maps, and shift compression pressure forward.

Cross-modal Information Fusion Compression

Use text information to guide visual token compression, enabling semantic-aware preservation of relevant information.

Token compression techniques have far-reaching significance for MLLM deployment:

• Mobile/edge computing scenarios: reduce latency and energy consumption;
• Long video/high-resolution document processing: support longer visual sequences;
• Commercial deployment: directly reduce inference costs.

Issues that still need to be explored:

1. How to preserve fine-grained spatial localization information during compression?
2. How to design task-adaptive compression strategies?
3. Can token compression for different modalities (images, videos, audio) be handled uniformly? These issues will drive the deepening development of the field.

Token compression is an important direction for MLLM development. By reducing visual token redundancy, it can significantly improve efficiency while maintaining performance. As the technology matures, we look forward to more efficient and deployable multimodal intelligent systems.