HIVE: Enhancing Multimodal Reasoning-Intensive Retrieval via Hypothesis-Driven Iterative Visual Evidence Retrieval

The HIVE (Hypothesis-Driven Iterative Visual Evidence Retrieval) framework injects explicit visual-text reasoning into the retriever through a four-stage process (initial retrieval, LLM-compensated query synthesis, secondary retrieval, LLM validation and re-ranking). It achieves an nDCG@10 of 41.7 on the MM-BRIGHT benchmark, 14.1 points higher than the best multimodal model, significantly improving the performance of multimodal reasoning-intensive retrieval.

In the field of information retrieval, multimodal queries (involving visual content like charts and screenshots and requiring deep text reasoning) are a challenge. Existing multimodal models perform poorly on the MM-BRIGHT benchmark (2803 real queries across 29 technical domains): the best multimodal model Nomic-Vision only achieves an nDCG@10 of 27.6, even lower than the pure text retriever DiVeR's 32.2 points, reflecting their defect in effectively integrating visual information and text logic.

HIVE is a plug-and-play framework consisting of four stages:

1. Initial Retrieval: Use a basic retriever to narrow down the range of candidate documents;
2. Compensatory Query Synthesis: LLM analyzes the visual/logical gaps in initial candidate documents and generates supplementary queries;
3. Secondary Retrieval: Use compensatory queries to obtain new candidate documents and fill in omissions;
4. Validation and Re-ranking: LLM verifies whether documents meet reasoning requirements and re-ranks them.

MM-BRIGHT evaluation results:

• Overall nDCG@10 reaches 41.7 (new SOTA);
• 9.5 points higher than the best pure text model DiVeR, and 14.1 points higher than the best multimodal model Nomic-Vision;
• The reasoning-enhanced retriever contributes 33.2 points, with an additional 8.5 points from the HIVE framework;
• Obvious advantages in domains with high visual demand: 68.2 points in games, 42.5 points in chemistry, and 49.4 points in sustainable development.

HIVE has plug-and-play characteristics and can work with various retrievers:

• Standard retrievers (traditional models without reasoning capabilities);
• Reasoning-enhanced retrievers (advanced models with certain reasoning capabilities); It is easy to integrate into existing systems and suitable for multiple scenarios.

HIVE reveals that retrieval is not just matching but reasoning. Traditional multimodal models implicitly handle visual-text associations and struggle in complex scenarios; HIVE uses explicit LLM intervention to externalize the reasoning process, with advantages of interpretability (outputs of each stage are traceable), controllability (optimizable by adjusting LLM prompts), and modularity (independent improvement of each stage).

HIVE technology is applicable to:

• Technical document retrieval (processing programming and engineering documents containing charts/screenshots);
• Academic literature search (integrating paper charts and main text);
• E-commerce product search (understanding the connection between images and specifications);
• Medical image retrieval (combining images with medical record text); As multimodal content grows, such deep understanding technologies will become more important.