HAWK: A New Breakthrough in Visual Token Pruning for Multimodal Large Models

HAWK proposes a visual token pruning method based on the importance of attention heads. It achieves 80% visual token pruning without training while maintaining 96% of the original accuracy, providing a feasible solution for the real-time deployment of multimodal large models.

Multimodal Large Language Models (MLLMs) are powerful, but high-resolution visual inputs lead to an explosive growth in the number of visual tokens, significantly increasing inference latency, computing resource consumption, and GPU memory usage. This makes it difficult to deploy them in real-time scenarios such as autonomous driving, robot control, or instant visual question answering on mobile devices.

Traditional visual token pruning assumes that all attention heads contribute equally, ignoring the functional differentiation of different attention heads in the Transformer architecture—some focus on shape contours, some on color textures, and others on spatial positional relationships. This leads to failure to consider the differentiated contributions of heads when evaluating token importance.

HAWK evaluates the importance of visual tokens through two dimensions: 1. Head importance weight: dynamically calculates the activation intensity and stability of each attention head to assign weights; 2. Text-guided attention: identifies relevant tokens in combination with the current text task. Moreover, it requires no training and can be directly applied to pre-trained MLLMs, making it plug-and-play.

In tests on the Qwen2.5-VL model, HAWK maintains 96.0% of the original accuracy after pruning 80.2% of visual tokens; the end-to-end inference time is reduced to 74.4% of the original; and GPU memory usage is significantly reduced, facilitating deployment on resource-constrained devices.

Technically, HAWK provides an effective path to solve the computational efficiency bottleneck of MLLMs and reveals the phenomenon of functional differentiation of attention heads; industrially, the training-free feature is suitable for fast-iterating engineering environments, enabling rapid evaluation of deployment optimizations and shortening the productization cycle.

Limitations: Currently, it only targets visual token pruning. Future directions: Expand to token optimization for other modalities; optimize the calculation of head importance weights; explore more aggressive compression schemes by combining technologies such as quantization and knowledge distillation.