GRIP: A New Feedback-Based Retrieval Method for Multimodal In-Context Examples

GRIP proposes a learnable retrieval framework based on model feedback. It identifies examples that truly improve ICL performance through contrastive training, addressing the limitations of traditional similarity-based retrieval in multimodal scenarios. It consistently outperforms similarity-based retrieval methods in image classification, image caption generation, and Visual Question Answering (VQA) tasks, and also has cross-model transferability.

When In-Context Learning (ICL) is extended to the multimodal domain, existing methods rely on selecting context examples from samples with semantically similar features in the feature space. However, studies have found that visually similar examples do not necessarily improve ICL performance. The core problem is how to identify examples that truly help improve model prediction quality rather than just similar ones.

GRIP (Guided Retrieval of In-context Prompts) no longer relies on static feature similarity. It introduces a learnable visual retrieval framework and uses feedback from Large Multimodal Models (LMMs) to judge the value of examples: those that guide the model to make accurate predictions are valuable examples, while others are detrimental.

GRIP uses a pure visual retrieval architecture and learns to distinguish between beneficial and harmful examples through contrastive training: it constructs positive examples that improve model performance and negative examples that reduce performance for the same query, going beyond visual similarity to understand the structure of examples that help solve tasks and continuously optimize retrieval strategies.

In image classification, image caption generation, and VQA tasks, GRIP outperforms similarity-based baselines on the Qwen2.5-VL-7B model; it shows significant gains in the Idefics2-8B classification task; and the trained retriever can be directly transferred to other models (including closed-source GPT-4o and Gemini) without retraining, reducing deployment costs.

In multimodal scenarios, visual similarity ≠ task relevance (e.g., similar images may belong to different categories or require different answers); ICL performance is affected by factors such as example diversity, order, and model knowledge, and simple feature similarity cannot capture these complex relationships; GRIP learns a more advanced 'task-aware' similarity.

It can be used to build multimodal RAG systems, visual assistants, or intelligent annotation tools, optimizing context example selection to improve system performance; its cross-model transferability allows one training to be reused across multiple underlying models, reducing deployment and maintenance costs.

GRIP breaks through the bottleneck of traditional similarity-based retrieval and provides new ideas for multimodal in-context learning; as large multimodal models develop, its feedback-driven methodology may inspire more research and promote the field toward a more intelligent and adaptive direction.