RESTORE: Correcting Position and Attention Distortions in Visual Token Compression for Multimodal Large Models

This article introduces the RESTORE framework, which targets position and attention consistency issues in visual token compression for multimodal large models. By calibrating attention weights and optimizing anchor selection strategies, it improves inference accuracy while maintaining computational efficiency. The framework is compatible with existing compression methods, requires zero additional training cost, and is a practical inference optimization solution.

Multimodal large models face the bottleneck of explosive visual token counts. The quadratic complexity of the Transformer attention mechanism leads to computational, memory, and latency issues. Existing visual token compression methods (merging, pruning, pyramid pooling) reduce costs but have blind spots: position distortion (inaccurate spatial position encoding) and attention distortion (distorted or lost attention patterns).

The RESTORE framework consists of two core components:

1. Attention Weight Calibration: Dynamically adjust attention weights based on token relative distances (enhance nearby tokens, attenuate distant ones), and introduce relative position encoding to restore position awareness.
2. Differentiated Anchor Selection: Evaluate token importance, ensure spatial diversity of anchors, and use differentiated weighting instead of simple averaging during merging to reduce information loss.

Experimental results show:

• Cross-method Improvement: Applied to mainstream compression methods like ToMe and Token Pooling, accuracy increases by 2-5 percentage points with zero additional training cost.
• Efficiency Preservation: Computational overhead increases by less than 5%, attention calibration has O(n) complexity, and anchor selection requires no additional forward propagation.
• Task Performance: Steady improvements in tasks like visual question answering (+2.8%), image captioning (+2.7%), and image-text retrieval (+3.1%).

The effectiveness is revealed through visualization and quantitative analysis:

• Attention Refocusing: After calibration, attention refocuses on semantically important regions; heatmaps show increased attention to key areas.
• Position Encoding Recovery: The distance-aware mechanism compensates for absolute position encoding distortion, leading to significant improvements in spatial reasoning tasks.
• Information Preservation: Feature similarity increases from 0.65 to 0.82, and differentiated anchor selection reduces information loss.

Deployment recommendations:

• Easy Integration: No need to modify model architecture, compatible with mainstream compression libraries like ToMe, and has few hyperparameters.
• Compression Rate Trade-off: For mobile devices, a compression rate ≤50% is recommended; for servers, it can reach up to 70%, with RESTORE restoring accuracy.
• Combined Optimization: Can be used in conjunction with techniques like quantization, speculative decoding, and cache optimization to improve overall efficiency.

Implications: Importance of joint compression-restoration optimization, transferability of attention patterns, and necessity of explicit position modeling. Limitations: Static compression strategy, lack of end-to-end training, insufficient research on cross-modal interaction. Future directions: Learnable calibration networks, content-aware dynamic compression, cross-modal alignment compensation methods.