A Survey of Token Compression Techniques in Multimodal Large Language Models: The Indispensable Path to Efficient MLLMs

This article surveys token compression techniques in Multimodal Large Language Models (MLLMs), focusing on how to improve model efficiency by reducing the number of visual tokens while maintaining multimodal understanding capabilities. With the development of MLLMs like GPT-4V and Gemini, the excessive number of visual tokens leads to high computational overhead and large memory requirements, limiting their application in resource-constrained environments. Token compression technology is the key to resolving this contradiction. This article will analyze from aspects such as background motivation, technical routes, representative models, experimental evaluation, and application directions.

Efficiency Bottlenecks of MLLMs

In traditional MLLMs, images are encoded into hundreds to thousands of visual tokens, which are input into the Transformer along with text tokens. The computational complexity grows in O(n²), leading to issues such as inference latency, high memory usage, and large training costs, limiting applications in resource-constrained scenarios.

Motivation and Challenges of Token Compression

Motivation: Reduce the number of visual tokens to lower computational overhead and improve efficiency. Challenges:

1. Information preservation: Reducing tokens without losing key visual details and semantics;
2. Cross-modal alignment: Compressed visual representations need to align with text semantics;
3. Task adaptability: Different tasks (e.g., image captioning, VQA) have different requirements for token granularity.

1. Spatial Aggregation Compression

• Spatial pooling: Adjacent patch features are merged via average/max pooling, which is simple and efficient but prone to losing fine-grained information;
• Clustering merging: e.g., ToMe, merging the most similar token pairs through similarity calculation.

2. Attention Mechanism Compression

• Importance sampling: Retain the Top-k tokens with the highest attention contribution, which has strong task adaptability;
• Query-aware compression: Dynamically determine the visual tokens that each text query needs to focus on.

3. Learned Compression Modules

• Learnable queries: Use learnable vectors to extract visual features (e.g., Perceiver architecture);
• MLP compressor: Map multiple tokens into one via a small MLP, learning non-linear strategies.

4. Multi-scale Hierarchical Compression

• Pyramid structure: Extract features at different resolutions—fewer tokens at higher levels for global representation, more tokens at lower levels for details;
• Dynamic resolution adjustment: Dynamically adjust the number of tokens based on content complexity.

Representative Models

• LLaVA-1.5: Uses a two-layer MLP projector to map 576 visual tokens into the language embedding space;
• Qwen-VL: Position-aware compression, pre-trained to adapt to various token counts;
• MiniGPT-4: Q-Former uses 32/64 learnable queries to extract visual features, significantly reducing the number of tokens;
• MobileVLM: Lightweight visual encoder and compression strategy, adapted for edge devices.

Experimental Insights

• Evaluation dimensions: Downstream task performance, compression ratio, inference speed, GPU memory usage, information retention;
• Key findings: Reducing 50%-80% of visual tokens only leads to a 1%-3% performance loss; Different tasks have different sensitivities (e.g., image-text retrieval is insensitive to compression, while fine-grained VQA requires more tokens).

Application Scenario Selection

• Cloud services: Lightweight compression, prioritizing performance;
• Edge devices: Aggressive compression + lightweight visual encoder;
• Real-time applications: Extremely low latency requirements, compression is a must.

Future Research Directions

1. Adaptive compression: Dynamically adjust the compression ratio based on input content;
2. Task-specific optimization: Customize compression strategies for downstream tasks;
3. Cross-modal joint compression: Jointly optimize text and visual redundancy;
4. Hardware-aware design: Optimize algorithms for NPU/TPU;
5. Video token compression: Extend to the temporal dimension to handle video tasks.

Token compression is a key technology for the practical application of MLLMs, solving the balance between efficiency and performance. From spatial pooling to learned modules, the technology is evolving rapidly. Future MLLMs will adopt more intelligent and adaptive compression strategies, making multimodal capabilities accessible to more device scenarios. Understanding and mastering these technologies is an important foundation for participating in the development of the next generation of multimodal AI.