GlimpsePrune: Dynamic Visual Token Pruning Technology for Large Vision-Language Models

The HVision-NKU team from Nankai University proposed GlimpsePrune, a dynamic visual token pruning method for large vision-language models (VLMs). Its core goal is to significantly improve inference efficiency while maintaining model performance by intelligently compressing visual information, addressing the deployment limitations of VLMs on edge devices and in real-time scenarios.

In recent years, VLMs such as GPT-4V, LLaVA, and Qwen-VL have achieved remarkable results in tasks like image understanding and visual question answering. However, processing high-resolution images requires a large number of visual tokens, leading to high inference latency and large memory usage, which severely limits their application on edge devices and in real-time scenarios.

The core of GlimpsePrune is dynamic visual token pruning, which differs from static pruning in that it adaptively retains important tokens based on the content of the input image. Specific strategies include: 1. Importance scoring (based on attention weights, gradients, or scoring networks); 2. Hierarchical pruning (progressive refinement of information); 3. Task awareness (adjusting strategies according to tasks). It also addresses three key challenges: balancing information retention and compression, controlling pruning computation overhead, and compatibility with existing models (plug-and-play module).

In VLMs, images are processed into visual tokens by visual encoders (e.g., ViT). A 224x224 image using 14x14 patches generates 256 tokens, and the number of tokens increases dramatically in high-resolution or multi-image scenarios. The self-attention complexity of Transformers is proportional to the square of the sequence length, so the growth of tokens leads to an explosive increase in computation and memory requirements.

This technology is applicable to: 1. Edge device deployment (mobile phones, IoT devices); 2. Real-time interactive applications (reducing response latency for visual question answering); 3. Batch image processing (saving time costs); 4. Multimodal large model services (reducing cloud computing costs and improving concurrency).

It is expected to reduce the number of visual tokens by 50% or more, while keeping the performance degradation within an acceptable range (e.g., within a few percentage points).

GlimpsePrune is an important progress in the field of VLM efficiency optimization, opening up new possibilities for the practical deployment of large models through dynamic token pruning. As multimodal AI becomes more popular, such efficiency optimization technologies will promote AI from the laboratory to a wider range of practical application scenarios.