LiteLVLM: Training-Free Visual Token Pruning for Accelerating Pixel-Level Localization Inference

Abstract: LiteLVLM proposes a training-free token pruning method based on CLIP reverse similarity, achieving 2.2x speedup and 2.3x memory savings while retaining 90% of the original performance, providing a new approach for efficient pixel-level localization in large vision-language models.

Core Idea: By reversing the visual-text similarity calculation of CLIP, strategically retain tokens critical for localization. It can be applied to existing pre-trained models without training, balancing efficiency and performance.

Research Background

In large vision-language models (LVLMs), visual tokens usually occupy the majority of the input sequence, leading to a significant increase in computational overhead. To alleviate this problem, recent studies have focused on pruning redundant visual tokens for image understanding tasks, but these methods perform poorly in pixel-level localization tasks—because the importance of tokens is highly dependent on the content of text input. How to effectively reduce computational burden without sacrificing localization accuracy has become a core problem to be solved in this field.

Core Finding: Counterintuitive Insight from CLIP

The research team discovered a counterintuitive phenomenon: Visual tokens within the target region often have lower similarity to the text. This subverts the traditional token importance evaluation思路—In pixel-level localization, tokens with low similarity to the text query may反而 contain key localization information.

LiteLVLM Technical Principles

LiteLVLM uses CLIP's cross-modal alignment feature for reverse filtering: Traditional methods retain tokens with high similarity to text, while LiteLVLM strategically retains tokens critical for localization and restores context tokens to achieve clear foreground-background separation, significantly reducing the number of tokens while maintaining precise perception of text-related regions.

Advantages of Training-Free

Unlike most optimization methods that require fine-tuning or retraining, LiteLVLM does not require any training or parameter updates. Users can directly apply it to existing pre-trained vision-language models without additional training data or computational resources. This plug-and-play feature greatly lowers the threshold for technical implementation and is more suitable for rapid deployment in actual production environments.

Benchmark Performance

Evaluated on pixel-level localization benchmarks such as the RefCOCO series, LiteLVLM significantly outperforms existing methods at all token compression ratios. When only 192 tokens are retained, it maintains performance close to the original model on the RefCOCO validation set.

Efficiency Improvement Metrics

• Inference Speed: 2.2x speedup, greatly reducing response time
• Memory Usage: 2.3x reduction in GPU memory consumption
• Performance Retention: Approximately 90% of the original model's performance

Cross-Model Compatibility

Validated based on mainstream pixel localization models such as GLaMM, proving good generality and transferability. The team provides a complete model repository and pre-trained weight download guide to facilitate community reproduction of the research.

Application Scenarios and Practical Value

Real-Time Interactive Applications

Suitable for low-latency response scenarios such as real-time image editing, intelligent annotation tools, and augmented reality systems. It allows large models that originally require high-end GPUs to be deployed on mobile devices or edge nodes.

Deployment in Resource-Constrained Environments

Provides a solution to reduce hardware costs for institutions/enterprises with limited computing resources, enabling more teams to access cutting-edge vision-language technology.

Multimodal System Optimization

As a key efficiency module in complex multimodal systems, it balances overall throughput and response quality.

Open-Source and Community Contribution

The project has been open-sourced on GitHub, providing a complete PyTorch implementation, detailed installation guide, and evaluation scripts. The code repository includes a one-stop toolchain from environment configuration to benchmark testing, supporting one-click reproduction of the paper's experimental results. It uses the Apache 2.0 license to encourage widespread use and improvement in academia and industry.

Technical Limitations and Future Directions

The current method is mainly optimized for pixel-level localization tasks, and its applicability to other visual understanding tasks needs further verification. In addition, maintaining stable performance under extreme compression ratios is also a future research direction. The team plans to continuously optimize the algorithm and explore integration schemes with more vision-language architectures.