G2TR: Generation-Guided Visual Token Compression Technology Boosts Efficiency of Multimodal Large Models

This article introduces G2TR, an innovative method for visual token compression using a generation-guided mechanism. It effectively reduces the computational overhead of unified multimodal models with separate encoders, significantly improving efficiency while maintaining model performance.

In recent years, unified multimodal models have adopted a separate encoder architecture, which preserves the independent representation capability of each modality but brings computational challenges: visual encoders generate a large number of tokens when processing high-resolution images, and the computational complexity grows quadratically when combined with text tokens, leading to inference latency and memory usage issues. Existing compression methods (clustering tends to lose fine-grained information, selection struggles to retain key information) find it difficult to balance compression and performance.

G2TR (Generation-Guided Visual Token Reduction) uses feedback signals from the generation process to guide visual token selection and compression. Its core idea is to enable the model to learn to identify important visual information during generation, aligning compression decisions with downstream task objectives and avoiding premature discarding of key information.

G2TR consists of four key components:

1. Generation-Aware Selection Module: Evaluates the importance of tokens for the generation task, considering the impact of future generation steps;
2. Dynamic Progressive Compression: Retains more tokens in early layers to capture global context, and gradually compresses redundancy in deeper layers;
3. Task-Adaptive Adjustment: Dynamically adjusts the compression level according to task requirements;
4. Separate Encoder-Friendly Design: Does not modify the pre-trained visual encoder, and introduces a compression module in the post-fusion layer to achieve plug-and-play functionality.

Experiments show that G2TR compresses 50%-70% of visual tokens while maintaining accuracy:

• Image Captioning Task: BLEU/CIDEr scores on the COCO dataset are comparable to the full model, with inference speed increased by about 40%;
• Visual Question Answering Task: VQA-v2 accuracy loss is <1%, and computational cost is significantly reduced;
• Generalization Ability: Stable performance across CLIP-ViT/DINOv2 encoders and language models of different scales.

G2TR provides important value for multimodal model deployment:

• Edge Devices: Enables high-end models to perform real-time inference on low-config hardware;
• Cloud Services: Improves concurrent processing capability and reduces service costs;
• Plug-and-Play Feature: Existing models can integrate optimization without retraining from scratch.

Currently, G2TR is mainly optimized for static images. Future explorations can include:

• Compression strategies for temporal visual content such as videos;
• Information loss issues under extreme compression ratios;
• Expansion to other modalities like audio token compression and long text sequence simplification.

G2TR is an important advancement in efficiency optimization for multimodal models. It balances performance and computational overhead through a generation-guided mechanism, clearing obstacles for the practical application of separate encoder models. As multimodal scenarios expand, such efficiency optimization technologies will become a key bridge for AI to move from the laboratory to industrial applications. We look forward to open-source code to facilitate more practices.