GeoBrowse: An Evaluation Benchmark for Geolocation Agents Combining Visual Reasoning and Multi-Hop Verification

This article introduces the GeoBrowse benchmark, which aims to evaluate the tool usage capabilities of multimodal agents. Combining visual clue combination and open web verification, this benchmark provides a new evaluation framework for in-depth research on agent development. Through geolocation tasks, GeoBrowse examines agents' ability to integrate multi-source information and use external knowledge for verification, filling the gap in existing evaluation benchmarks regarding the combination of visual and multi-hop reasoning.

With the rise of in-depth research on agents, evaluating their capabilities faces challenges. Existing benchmarks have limitations: BrowseComp focuses on pure text multi-hop reasoning but lacks visual assessment, while multimodal benchmarks rarely require both weak visual clue combination and multi-hop verification. Geolocation is an ideal evaluation scenario because it requires integration of multi-source visual clues (e.g., architectural style, road signs), open knowledge verification (e.g., searching text information to confirm locations), and is based on real-world scenarios, which can reflect the actual performance of agents.

The GeoBrowse benchmark has two difficulty levels: Level 1 tests visual clue extraction and combination capabilities; Level 2 introduces long-tail knowledge and entity confusion to test deep reasoning. The supporting GATE agent workflow includes 9 tools: 5 image thinking tools (e.g., detail observation, text recognition) for visual analysis, and 4 knowledge-intensive tools (e.g., fact verification, geographic query) for web retrieval. Additionally, expert-annotated reasoning trajectories are provided to support trajectory-level analysis.

Experiments show that GATE significantly outperforms direct reasoning and open-source baselines. Key findings: Pure perception or pure search is insufficient; visual and knowledge integration is needed; collaborative use of tool combinations is more effective than single tools; GATE's advantages come from coherent domain-specific planning (prioritizing image tools in Level1, actively calling knowledge tools in Level2), achieving key evidence steps, and fewer errors in the integration phase. Moreover, performance improvement stems from reasonable tool usage planning, not just increasing the number of calls.

GeoBrowse is an important advancement in multimodal agent evaluation, combining visual reasoning and knowledge verification to provide a rigorous assessment framework. Core findings emphasize the importance of tool usage strategies: successful agents need appropriate tools and correct timing of use. GATE's performance demonstrates the value of coherent, task-aware tool usage planning. Such benchmarks will help understand the capability boundaries of agents and guide the development of next-generation agents.

Insights from GeoBrowse for agent research: Evaluation benchmarks should focus on multimodal integration, open verification, and fine-grained analysis; tool usage strategies need to dynamically adapt to tasks, be evidence-oriented, and improve integration capabilities. Limitations include domain specificity, language bias (mainly English resources), and toolset constraints. Future directions can include expanding to other domains, developing intelligent tool selection algorithms, and researching the value of multi-agent collaboration.