TwigVLM: An Innovative Method to Accelerate Large Vision-Language Models via Model Pruning

Large Vision-Language Models (LVLMs) are powerful in multimodal tasks, but their massive scale leads to high inference costs. The ICCV 2025 paper TwigVLM proposes the "growing twigs" structured pruning methodology, which significantly improves inference speed while maintaining over 95% of the original performance, providing a key solution for LVLMs to move towards practical applications.

LVLMs have four major computational bottlenecks: high overhead of visual encoders, high complexity of projection layers, large scale of language models, and sequence length inflation caused by visual tokens. Traditional compression methods (pruning, quantization, distillation) require retraining from scratch or extensive fine-tuning, which are costly. Researchers need lightweight and flexible acceleration solutions for pre-trained models.

Core idea: Identify low-contribution parameters/computational paths in LVLMs, and achieve acceleration through structured pruning with minimal performance loss. Differences from traditional pruning: 1. Structured pruning (removing entire functional modules/paths); 2. Task-aware (considering downstream task requirements); 3. Progressive optimization (iteratively exploring optimal structures); 4. Recoverability (supporting dynamic adjustment of model scale).

1. Contribution evaluation mechanism: Activation analysis, gradient sensitivity, task performance ablation experiments, attention pattern analysis; 2. Modular pruning strategy: Visual encoder optimization (identifying key feature subsets), projection layer compression (low-rank decomposition/sparsification), language model adaptation (attention optimization to reduce visual token redundancy); 3. Dynamic adjustment capability: Automatically select configurations based on input complexity/task requirements (e.g., lightweight version for simple images, full version for complex scenes).

Maintained over 95% of original performance on benchmark tests such as image captioning (COCO Captioning), visual question answering (VQA/GQA), image-text retrieval (Flickr30K/COCO Retrieval), and multimodal reasoning (ScienceQA); Efficiency improvements: Inference latency reduced by 30-50%, memory usage decreased, throughput increased, energy consumption lowered; Cross-model generalization: Effective on multiple LVLM architectures.

1. Edge device deployment: Suitable for mobile/embedded systems, expanding LVLM application scenarios; 2. Real-time interactive applications: Low latency supports real-time visual assistants and interactive image editing; 3. Cost optimization: Cloud deployment reduces computational resource investment and lowers operational costs.

Current limitations: Task specificity (cross-task migration needs research), precision trade-off (caution required for high-precision scenarios), dynamic adjustment overhead; Future directions: Automated pruning tools, end-to-end joint optimization, hardware co-design (for NPU/TPU).