INTERLACE: Efficient Layer Pruning and Adaptive Techniques for Vision-Language Models

This article introduces the INTERLACE method accepted by CVPR 2026, developed and open-sourced on GitHub by pmadinei (link: https://github.com/pmadinei/Interlace). This method significantly reduces computational costs while maintaining the performance of vision-language models (VLMs) through interleaved layer pruning and efficient adaptive techniques, aiming to solve the efficiency dilemma of VLMs.

Vision-language models (such as CLIP, LLaVA, GPT-4V) have reshaped the boundaries of AI, but face efficiency challenges:

• Billions of parameters require massive computing resources
• Inference latency limits real-time applications
• Deployment costs hinder widespread adoption
• Energy consumption restricts deployment on edge devices How to improve efficiency while maintaining capabilities has become a key issue in the VLM field.

Interleaved Layer Pruning Strategy

• Interleaved Layer Retention Mechanism: Analyze the contribution of layers to vision-language alignment, selectively retain key layers, remove redundant layers, and ensure multi-scale feature capture
• Progressive Pruning: Dynamically adjust layer importance evaluation in multiple stages

Efficient Adaptive Techniques

• Residual connection reorganization: Compensate for information loss from pruning
• Attention head reallocation: Optimize attention efficiency of remaining layers
• Feature distillation: Use the original model to guide the learning of the pruned model

Technical Details

• Layer Importance Evaluation: Multi-dimensional metrics including gradient sensitivity, feature similarity, and task relevance
• Joint Pruning-Fine-tuning Optimization: Alternate pruning and parameter updates, introducing sparse regularization
• Multimodal Feature Alignment: Protect hierarchical features of visual encoders, text semantic representations, and cross-modal projection layers

Experimental Results

• Parameter count reduced by 30-50% (maintaining over 90% performance)
• Inference speed increased by 1.5-2 times
• Downstream task performance: Image captioning retains over 95% CIDEr score, VQA accuracy drops within 3%, and image-text retrieval Recall@K remains at a high level
• Cross-model transfer: Applicable to various VLMs such as CLIP, BLIP, LLaVA

Application Scenarios

• Mobile Devices: Real-time image captioning, smart albums, AR applications
• Edge Computing: Intelligent monitoring, industrial quality inspection, retail analysis
• Cloud Services: Reduce inference costs, lower energy consumption, and improve response speed

Comparison with Traditional Methods

• Magnitude pruning: Simple but with limited effect
• Structured pruning: Hardware-friendly but aggressive
• Knowledge distillation: High training cost

Advantages of INTERLACE

• Designed for VLM characteristics
• Joint optimization reduces training overhead
• Strong multi-task generalization ability
• Concise and efficient engineering implementation

Limitations and Future Directions

• Limitations: Performance cliff with excessive pruning, task-specific differences, limited adaptability to dynamic scenarios
• Future directions: Automated pruning ratio, dynamic pruning, integration with NAS, hardware-aware pruning

INTERLACE promotes efficiency optimization of VLMs:

• Academia: Provides a new methodology for VLM compression
• Industry: Lowers deployment thresholds and accelerates implementation
• Green AI: Reduces computing resource consumption
• Inclusive AI: Enables more users to access VLM capabilities

This work combines academic innovation and engineering value, driving VLMs toward efficiency and inclusiveness, and is an important enabler for balancing capability and efficiency in future VLMs.