ZipRerank: Efficient Multimodal List Reranking for Long Documents

Researchers propose ZipRerank, which addresses the efficiency bottleneck of multimodal reranking for long documents. Through a lightweight query-image early interaction mechanism and a single forward pass scoring strategy, it reduces LLM inference latency by an order of magnitude. It achieves or surpasses the performance of SOTA multimodal rerankers on the MMDocIR benchmark and is suitable for latency-sensitive real-time systems.

In vision-centric retrieval and M-RAG systems, list reranking is a key component, but traditional methods face two major bottlenecks: 1. Excessively long visual token sequences lead to a surge in computational overhead; 2. Multi-step autoregressive decoding limits throughput, making SOTA VLM-based rerankers difficult to meet the needs of latency-sensitive systems.

ZipRerank aims to balance efficiency and accuracy, with two major innovations: 1. Lightweight query-image early interaction: Let queries interact with image features early in visual encoding, focus on relevant regions, and compress the length of visual tokens; 2. Single forward pass scoring: Process all candidate documents in parallel, capture list-level relationships, directly optimize the ranking objective, and reduce time complexity from O(n × number of decoding steps) to O(1).

ZipRerank adopts a two-stage training approach: 1. List-level pre-training: Pre-train on large-scale text-rendered images to learn basic reranking capabilities; 2. Multimodal fine-tuning: Use VLMs like GPT-4V as teacher models to generate soft ranking signals, combined with real data for fine-tuning, and the loss design considers both ranking and distillation losses.

On the MMDocIR benchmark, ZipRerank achieves or surpasses SOTA accuracy; in terms of efficiency, inference latency is reduced by an order of magnitude (about 10x), throughput is improved, resources are saved, making it suitable for real-time search, high-concurrency services, resource-constrained environments, and streaming processing scenarios.

Limitations include possible loss of fine-grained information due to compression, differences in domain adaptation, and dependence on teacher model biases; future directions: more aggressive compression strategies, teacher-free/less-teacher training, expansion to video modalities, adaptive interaction mechanisms, and joint optimization of retrieval components.

Implications: Multimodal RAG needs to emphasize efficiency optimization, design task-specific lightweight architectures, use knowledge distillation, and focus on list-level modeling; Conclusion: ZipRerank proves that efficiency and accuracy can be achieved simultaneously, providing a reference for the design of multimodal retrieval systems, and efficient reranking technology will become more important in the future.