DixitWorld: Evaluating the Abductive Reasoning Ability of Multimodal Vision-Language Models Using a Board Game

DixitWorld is an open-source project from an ACL 2026 paper. It constructs a multi-agent benchmark framework using the classic board game Dixit, aiming to evaluate the abductive reasoning ability of Vision-Language Models (VLMs) and reveal the structural asymmetry in hypothesis generation and selection tasks among current VLMs. The project includes the dynamic game environment DixitArena and the static dataset DixitBench, providing new tools for evaluating the high-order cognitive abilities of VLMs.

Abductive reasoning is the ability to generate explanatory hypotheses from partial observations, which is a weak point in the 'understanding' aspect of current large models. Traditional VLM benchmarks are mostly static and lack evaluation of creative hypothesis generation and pragmatic reasoning. DixitWorld transforms the board game into a dynamic multi-agent game scenario, simulating human analogical reasoning and creative thinking, thus filling this evaluation gap.

1. DixitArena: A dynamic multi-agent game environment where agents take turns playing the roles of Storyteller (generating moderately ambiguous clues, corresponding to hypothesis generation) and Listener (selecting the target image based on clues, corresponding to hypothesis selection). The scoring mechanism requires models to master pragmatic skills.
2. DixitBench: A static multiple-choice question dataset containing 84 images and 3 difficulty levels. Distractors are controlled based on semantic similarity, and the correlation with the results of the DixitArena Listener task reaches Pearson r=0.947.

Evaluation of 6 mainstream VLMs (such as Qwen2.5-VL, GPT-4o, etc.) found:

• Storyteller task: Over 78% of rounds scored zero, as models struggle to balance the ambiguity and comprehensibility of clues, reflecting a deficit in pragmatic control;
• Listener task: The best model achieved an accuracy of approximately 75.6%, showing strong discriminator capabilities. This reveals a significant ability gap between the 'generation' and 'selection' directions of VLMs.

The project uses a Python modular design:

• Core code: src/game.py (game engine), src/agents.py (agent definition), src/call_api_*.py (multi-API clients);
• Experiment scripts: experiments/ supports Arena tournaments, batch testing, and DixitBench evaluation;
• Analysis tools: analysis/ provides statistics and visualization. It supports API key configuration for OpenRouter, Together, etc., facilitating reproduction and expansion.

DixitWorld has been recognized by ACL 2026, and its value includes:

1. Filling the gap in existing benchmarks for evaluating creative generation and pragmatic reasoning;
2. Providing a multi-agent collaboration/competition experimental platform to study communication games and emergent behaviors;
3. Guiding AI applications: Human supervision remains indispensable in scenarios such as education and creative writing.

Limitations: The 84 Dixit cards are not included due to copyright issues, increasing the threshold for reproduction; only English context is evaluated. Future directions: Improve Storyteller task performance through fine-tuning/prompt engineering; expand cross-language evaluation; optimize dataset copyright issues.

DixitWorld represents an important step in AI benchmarks towards exploring higher-order cognitive abilities. Recognizing objects is just the beginning for models; the real challenge is understanding expressions like 'as mysterious as a cat'—balancing imagery association and interpretive space, which is a key threshold for naturalizing human-computer interaction.