LightKV: Making KV Caching Lighter for Large Vision-Language Models

LightKV addresses the memory bottleneck of KV caching in Large Vision-Language Models (LVLMs) using a cross-modal message-passing mechanism to compress the KV cache of visual tokens. By retaining only 55% of the original visual tokens, it halves the KV cache size, reduces computational load by 40%, maintains model performance, and significantly outperforms compression baselines that only consider visual information.

Large Vision-Language Models (LVLMs) can understand images and answer visual questions, but GPU memory consumption is a deployment bottleneck. KV cache is a key component in Transformer inference; its memory overhead is manageable in pure-text LLMs, but in LVLMs, visual encoders generate a large number of visual tokens (hundreds or even thousands for high-resolution images), leading to a sharp increase in memory during the pre-filling phase when their KV representations are cached.

Redundancy of Visual Tokens

Visual token embedding vectors have significant redundancy: adjacent image patches have similar information, background regions contribute little, and some features are semantically repetitive—there's no need to retain all of them with full precision.

Text-Guided Compression

Unlike pure visual compression, LightKV introduces text guidance: the purpose of visual tokens is to respond to text prompts, so cross-modal message passing is used to let text prompts guide compression, ensuring that the visual tokens most relevant to the current task are retained.

Cross-Modal Message Passing

1. Initial Representation: The visual encoder generates visual token embeddings, and the text encoder generates text representations.
2. Message Aggregation: Visual tokens aggregate information from other tokens based on their relevance to the text prompt to identify key information.
3. Progressive Compression: Gradually compress during the pre-filling phase—retain/aggregate relevant tokens, merge/discard redundant ones.

This mechanism dynamically adapts to different text prompts; the same image will have different compression patterns for different tasks.

Comparison with Pure Visual Compression

Method Type	Decision Basis	Limitation
Pure Visual Compression	Spatial position, visual features	Ignores task relevance, may discard key information
LightKV	Cross-modal text-visual relevance	Prompt-aware, retains task-relevant information

For example, in complex scenes, pure visual methods compress uniformly, while LightKV focuses on relevant objects based on the text question.

Evaluation Setup

Evaluated on 8 open-source LVLMs and 8 benchmark datasets (including MME, SeedBench, etc., covering tasks like visual question answering, image captioning, OCR).

Key Results

1. Halved KV Cache: The KV cache size of visual tokens is reduced by 50%, lowering hardware requirements.
2. 40% Less Computational Load: The complexity of attention computation is related to the square of the number of tokens; compression improves inference speed.
3. Performance Retention: Maintains the original model's capabilities across all benchmark tests; some tasks see slight improvements due to noise reduction.
4. Outperforms Existing Baselines: The text-guided strategy is superior to pure visual compression methods.

1. Edge Deployment Possibility: Reduces memory overhead, enabling LVLMs to be deployed on mobile devices, embedded systems, and IoT devices.
2. Long Video Understanding: Compresses visual tokens to support processing longer video clips.
3. Multi-Image Reasoning: Reduces memory pressure from visual tokens of multiple images, making tasks like comparing differences more feasible.

Current Limitations

1. Upper Limit of Compression Ratio: Retaining 55% of tokens is the current balance; more aggressive compression may affect performance.
2. Computational Overhead: Cross-modal message passing requires additional computation, but this is offset by subsequent attention savings.
3. Generalization: Needs validation on more LVLM architectures.

Future Directions

• Adaptive Compression Rate: Dynamically adjust based on image complexity and task.
• Hierarchical Compression: Multi-granularity (patch/region/object level) compression.
• Combination with Quantization: Further enhance memory savings.

Implications for LVLM Architecture Design

1. Depth of Modal Interaction: Compressing visual information requires language guidance; prompt design should consider deep modal fusion.
2. Unification of Efficiency and Effectiveness: LightKV proves that compression can achieve both efficiency and effectiveness, even improving performance through noise reduction.
3. Value of Dynamic Inference: Adaptively allocating resources based on input is key to efficient AI systems.

Conclusion

LightKV provides an elegant solution for LVLM memory efficiency, achieving significant optimization via text-guided cross-modal compression. It reveals the trend of blurring modal boundaries in multi-modal AI and offers insights for designing efficient multi-modal systems.