A New Method for Visual Evidence Selection in Multimodal RAG: Paradigm Shift from Semantic Relevance to Information Gain

This paper proposes an information theory-based visual evidence selection framework for multimodal Retrieval-Augmented Generation (RAG). By defining evidence utility as the information gain on the model's output distribution, it solves the utility mismatch problem caused by traditional methods' reliance on semantic relevance. The framework uses a lightweight proxy model to efficiently estimate evidence utility, achieving dual optimization of performance improvement and computational cost reduction.

In multimodal RAG systems, visual evidence selection directly affects answer quality. Existing methods rely on semantic relevance or surface similarity to select evidence, but these metrics often have significant mismatches with the actual utility for downstream reasoning. For example, when querying architectural styles, the system may retrieve semantically relevant building images but lack key visual features to judge the style, creating a gap where 'relevance ≠ utility'.

The research team reformalized the evidence selection problem from an information theory perspective, defining evidence utility as information gain (the change in information quantity of the model's output distribution due to evidence), which directly aligns with reasoning goals. To address the computational infeasibility of optimizing over the answer space, they introduced the concept of 'evidence usefulness at the latent variable level' and proved its equivalence to the utility ranking in the answer space, laying the foundation for efficient algorithm design.

The core of this method is using a lightweight multimodal model as a 'utility predictor' to capture the complex relationship between evidence and reasoning goals. Through precomputation and caching mechanisms, it quickly evaluates the utility scores of a large number of candidate visual evidence without running full large model inference, balancing theoretical rigor and deployment efficiency.

On the authoritative benchmarks MRAG-Bench and Visual-RAG, this method consistently outperforms existing state-of-the-art RAG baselines while significantly reducing computational costs. This means that in practical deployment, better answer quality and faster response speed can be achieved simultaneously, especially suitable for resource-constrained scenarios.

This work provides practitioners with a clear theoretical framework to help understand evidence value; the lightweight proxy design is easy to integrate into existing RAG pipelines without large-scale retraining. For image-intensive scenarios (such as medical image analysis, industrial quality inspection, and visual question answering), the utility-oriented selection strategy can improve the experience.

This research marks the paradigm shift of multimodal RAG from 'relevance-driven' to 'utility-driven', providing both precise evidence selection criteria and computational efficiency. With the deployment of multimodal large models, this method offers a theoretical foundation and practical tools for the efficient use of visual information, driving the next generation of systems toward more intelligent evolution.