DR-MMSearchAgent: Deepening the Reasoning Capabilities of Multimodal Search Agents

DR-MMSearchAgent addresses the premature interaction collapse problem of multimodal search agents by proposing two innovative mechanisms: trajectory-level advantage estimation based on structural proximity and dynamic calibration of differential Gaussian rewards. These effectively incentivize agents to fully explore information, outperforming the baseline MMSearch-R1 by 8.4% on FVQA-test and significantly enhancing the reasoning capabilities of multimodal search agents.

Multimodal search agents often encounter premature interaction collapse: they terminate interactions before fully collecting information and directly give potentially incorrect answers. The root causes include two points: 1. Limitations of terminal rewards—failure to distinguish exploration behaviors, suppression of exploration motivation, and neglect of process quality; 2. Redundant context overwhelming feedback—massive redundant information from multi-round interactions makes it difficult to extract key signals. These two factors reinforce each other, leading agents to fall into local optima of shallow interactions.

DR-MMSearchAgent has two core innovations:

1. Trajectory-level advantage estimation based on structural proximity: Derive advantage signals from the entire trajectory rollout. By comparing the exploration sufficiency of structurally similar trajectories in the same batch, higher advantages are given to deeply explored trajectories to incentivize full interaction;
2. Dynamic calibration of differential Gaussian rewards: Maintain dynamic interaction tolerance parameters (adjusted based on context redundancy, information gain, and answer confidence). Use a Gaussian reward function—encourage exploration when tolerance is high, and convergence when tolerance is low—to suppress redundant searches and adaptively adjust search depth.

Dataset Construction: Build a multi-step deep reasoning dataset containing 3602 high-quality question-answer pairs. Each question requires at least 3 steps of reasoning, with annotations for standard reasoning paths, key information points, interfering information, and tool call sequences; Experimental Results: On FVQA-test, DR-MMSearchAgent achieves 67.5%, an 8.4% improvement over the baseline MMSearch-R1 (62.3%). Ablation experiments show that trajectory-level advantage estimation contributes +4.2% and differential Gaussian rewards contribute +3.1%, with better combined effects. Interaction analysis indicates that its average number of interaction rounds (4.1 rounds), information sufficiency score (8.7/10) are higher than the baseline, and the redundancy rate (12%) is significantly reduced.

Reasons for DR-MMSearchAgent's effectiveness:

1. Improvement in advantage estimation: Advantage signals are strongly correlated with exploration depth (correlation coefficient 0.78), truly reflecting the value of exploration;
2. Reward shaping effect: Rewards grow gradually with information collection, avoiding early saturation and encouraging continuous exploration;
3. Change in attention pattern: It can effectively focus on key information and reduce attention dispersion on redundant content.

Implications of DR-MMSearchAgent for agent research:

1. Reward design should focus on process-level rewards and trajectory-level evaluation to break through the limitations of terminal rewards;
2. Automatically adjust exploration depth through reward design instead of relying on fixed exploration strategies;
3. Adaptive context management (such as Gaussian reward mechanism) is crucial for handling long contexts.

Limitations: High computational overhead of trajectory-level advantage estimation, Gaussian reward parameters requiring task tuning, and generalization to be verified; Future Directions: Develop efficient trajectory-level advantage estimation algorithms, meta-learn adaptive parameters, verify generalization through multi-task training, and conduct in-depth theoretical analysis of the structural proximity hypothesis.