LiteLVLM: Training-Free Efficient Token Pruning for Vision-Language Models

The LiteLVLM technique proposed by the Computer Vision Research Group at Sejong University achieves efficient token pruning without any training or fine-tuning by reversing CLIP's vision-text similarity mechanism. This technique retains approximately 90% of the performance in pixel-level localization tasks while achieving a 2.2x inference speedup and 2.3x memory optimization, providing a new solution for the efficient deployment of large vision-language models.

In large vision-language models (LVLMs), visual tokens account for over 80% of the input sequence, leading to severe computational bottlenecks. Existing token pruning methods are mostly designed for image understanding tasks, sorting based on the intrinsic importance of visual features, but perform poorly in pixel-level localization tasks—since the importance of the same visual region varies greatly across different text queries, traditional strategies ignore text-guided key information.

The core insight of LiteLVLM comes from CLIP analysis: visual tokens within the target object region often have low CLIP similarity to the text query. This is because CLIP matches text based on global image features, while the semantics of local tokens differ from the global representation; even though target region tokens contain key information, their similarity is conversely low.

LiteLVLM adopts a two-stage pruning process: 1. Retain visual tokens with low CLIP similarity to the text query (corresponding to the target region); 2. Restore some context tokens to solve the foreground-background boundary ambiguity problem and ensure the preservation of fine-grained spatial information.

Evaluations on referential expression segmentation datasets such as RefCOCO and RefCOCO+ show: when only 192 tokens are retained, approximately 90% of the original model's localization accuracy is maintained; inference speed is increased by 2.2x; GPU memory usage is reduced by 2.3x; and it requires no training or fine-tuning at all, making it directly applicable to any CLIP-based LVLM.

Technical contributions: Reveal the intrinsic characteristics of CLIP in fine-grained localization tasks, providing new ideas for LVLM design. Application scenarios: Edge device deployment, real-time interactive systems, cloud cost optimization, multimodal AI Agent backends, etc.

LiteLVLM solves the visual token redundancy problem with a concise design, and its "reverse CLIP similarity" idea is counterintuitive and profound. As a paper accepted by ICML 2026, it represents the latest progress in the field of efficient multimodal reasoning, and will play an important role in reducing deployment costs and expanding the application scope of LVLMs in the future.