QUACK: The First Multimodal Social Reasoning Evaluation Benchmark for Vision-Language Models

QUACK (Questioning, Understanding, and Assessing Collaborative Knowledge) is the first multimodal social reasoning evaluation benchmark designed specifically for vision-language models (VLMs), built on a fully open-source engine. It fills the gap in traditional text-only evaluations, assessing models' spatial reasoning, social reasoning, and deception detection capabilities through mechanisms like graph-structured map navigation, limited field-of-view observation, and multi-round discussion and voting. It supports multi-model comparison experiments and a reproducible evaluation environment.

Current large language model evaluations are mostly limited to text-only scenarios, making it difficult to fully assess the multimodal perception, spatial navigation, and social reasoning capabilities required for real-world intelligent agents. Traditional social reasoning game benchmarks (e.g., Werewolf) have three major limitations: lack of spatial grounding (inability to verify location authenticity), inability to test visual understanding, and limited partial observability. QUACK introduces a spatial dimension to address these issues, allowing agents to interact in an environment similar to 'Among Us'.

The core of QUACK is a configurable graph-structured map system that uses weighted corridors to connect rooms, simulating real spatial relationships. Key design features include: limited field of view (only agents in the same room are visible), multi-tick location-bound tasks (requiring staying in a specific location for multiple steps to complete), emergency meeting mechanism (multi-round discussion and voting), and structured state input (global map + local perception + text state). These designs force models to perform long-range multimodal reasoning, integrating multi-source information such as historical trajectories and task progress.

QUACK's evaluation protocol has three layers:

1. Basic Task Performance: Task completion rate (efficiency of regular agents in completing tasks), survival rate (probability of survival in the presence of impostors), win rate (team victory probability);
2. Social Coordination Ability: Meeting participation, voting accuracy (identifying impostors), persuasiveness (success rate of impostors in misleading others);
3. Adversarial Robustness and Behavioral Consistency: Through an automatic statement verification pipeline, detect deceptive behaviors, evaluate belief consistency, and audit action-speech alignment to achieve fine-grained evaluation.

QUACK supports multiple mainstream VLMs (GPT-5.2/GPT-5.4, Claude Opus4.6, Gemini3.1 Pro, Grok4, Kimi K2.5). It allows running homogeneous/heterogeneous experiments (e.g., GPT-5.2 Geese vs Claude Opus4.6 Ducks) via command-line parameters and provides batch experiment scripts. Experimental reproducibility is ensured by recording random seeds, complete decision sequences, and rendered frames/videos, supporting game replay from logs.

QUACK is developed in Python, using Hydra for hierarchical configuration management. Configuration files are composable YAML (main entry, game rules, map definitions, model settings, etc.). Game logs are saved in JSONL format, including full state, decisions, meeting records, etc. Evaluation scripts support multi-level analysis, and replay scripts can generate rendered frames or videos.

QUACK is not only an evaluation tool but also a research platform that reveals core issues:

• Multimodal Grounding: Testing models' real understanding of spatial relationships;
• Deception and Anti-Deception: Exploring AI's deception capabilities, lie detection, and trust mechanisms;
• Long-Range Memory Reasoning: Assessing models' ability to maintain long-term behavioral memory and use it for reasoning;
• Multi-Agent Collaboration and Competition: Studying collaboration differences and emergent behaviors among different models.

QUACK represents an important step forward in AI evaluation toward real-world complex scenarios. It is an experimental environment for systematically exploring the boundaries of VLMs' spatial reasoning, social intelligence, and strategic behavior capabilities. It helps understand models' real capabilities, identify limitations, and guide research directions. Researchers focusing on AI agents, multimodal reasoning, and social intelligence are recommended to explore the QUACK tool in depth.