FIKA-Bench: A New Benchmark for Fine-Grained Knowledge Acquisition Capabilities of Multimodal Agents

FIKA-Bench is a new benchmark targeting the fine-grained knowledge acquisition capabilities of large multimodal models and agents, consisting of 311 real-world scenario instances. The study found that the accuracy of the current state-of-the-art systems is only 25.1%, revealing that combining fine-grained visual recognition with external knowledge retrieval remains a significant challenge.

Existing multimodal benchmarks mainly focus on visual recognition itself and lack systematic evaluation of systems' ability to actively acquire external knowledge. Large multimodal models (LMMs) have made progress in general visual understanding, but they struggle with fine-grained tasks that require combining visual details with external knowledge (e.g., distinguishing similar bird species, identifying specific architectural styles).

FIKA-Bench fills the gap in evaluation, consisting of 311 real-world instances, with three key features:

1. Leakage Prevention Design: Samples are filtered through closed-book models to ensure they are not memorized by models, forcing reliance on external retrieval;
2. Evidence Anchoring: All samples are supported by verifiable evidence, and answers can be verified in external resources;
3. Fine-Grained Challenges: Covers high-precision scenarios such as distinguishing similar species and identifying subtle differences.

Evaluations of the latest multimodal models and agents show that the accuracy of the best system is only 25.1%, with no model exceeding 30%. Simply equipping tools (e.g., search engines) is not enough to bridge the gap; effective use of tools remains a challenge.

Core reasons for agent failure:

1. Entity Retrieval Errors: Misunderstanding visual content leads to deviated retrieval queries, or failure to identify key entities;
2. Insufficient Visual Judgment: Inability to accurately compare retrieved information with visual evidence, making it difficult to select the correct answer. Fundamental improvements to agent design are needed to focus on fine-grained visual recognition.

Future agents need:

1. Stronger visual understanding capabilities to identify subtle differences and convert them into effective retrieval queries;
2. Better evidence evaluation mechanisms to integrate retrieval results with visual evidence;
3. Multi-stage reasoning (initial observation → hypothesis → retrieval verification → correction → re-verification). Interactive methods similar to human cognition may break through the bottleneck.

FIKA-Bench marks a new stage in multimodal AI evaluation, providing standardized tools and revealing technical limitations. The 25.1% accuracy rate indicates that there is still a long way to go to build human-level knowledge acquisition agents. This benchmark will inspire researchers to explore new architectures, training methods, and evaluation paradigms, promoting more reliable and practical multimodal agents.