WikiVQABench: A New Knowledge-Driven Visual Question Answering Benchmark to Test Multimodal Models' External Knowledge Reasoning Ability

WikiVQABench is a knowledge-driven Visual Question Answering (VQA) benchmark built on Wikipedia and Wikidata, designed to evaluate the performance of Vision-Language Models (VLMs) in scenarios requiring external knowledge reasoning. This benchmark fills the gap where traditional VQA benchmarks overlook the need for knowledge-intensive reasoning. By integrating images, article titles, and structured knowledge, it provides a more comprehensive perspective for assessing the capabilities of multimodal models.

Traditional VQA benchmarks mainly focus on perceptual tasks (which can be answered directly from image content), but many questions in real-world scenarios require external knowledge to answer (e.g., the city where the Eiffel Tower is located requires geographical knowledge). Existing VQA benchmarks ignore such knowledge-intensive reasoning needs, leading to overestimation of VLMs' performance in real applications.

WikiVQABench integrates three types of data sources: Wikipedia images, article titles, and Wikidata structured knowledge. It automatically generates image-question-answer combinations via LLMs, then filters them through human review to ensure factual accuracy, visual-text consistency, and that questions require combining external knowledge with visual evidence to answer—thus ensuring the benchmark's high quality.

Evaluation of 15 VLMs with parameter sizes ranging from 256 million to 90 billion shows: accuracy ranges from 24.7% to 75.6%; larger models usually perform better but non-linearly; even the largest models still have room for improvement in complex knowledge reasoning problems. This benchmark effectively distinguishes models' knowledge-intensive reasoning capabilities.

WikiVQABench provides researchers with a standardized evaluation tool, emphasizing that VLMs need to have knowledge integration capabilities (understanding images + grasping the world knowledge behind them). Its testing capabilities are crucial for scenarios such as intelligent education, museum navigation, medical image analysis, and autonomous driving.

The WikiVQABench dataset and evaluation code have been made public and can be obtained via the paper address (http://arxiv.org/abs/2605.21479v1). The open-source code is provided along with the paper. Openness ensures that the community can continuously improve evaluation methods and track the progress of VLMs' knowledge reasoning.

Future exploration directions for WikiVQABench include: multilingual expansion (currently based on English Wikipedia), dynamic knowledge updates (synchronizing the latest knowledge base information), fine-grained analysis (models' performance across different knowledge types), and knowledge injection methods (effectively integrating into VLMs' pre-training/fine-tuning).