CRAFT Benchmark: Multi-Agent Coordination Remains an Unsolved Problem—Strong Reasoning ≠ Good Collaboration

The CRAFT benchmark requires multiple agents to collaboratively build 3D structures under incomplete information. Tests show that stronger reasoning ability does not translate into better coordination, and small models can often match or even outperform cutting-edge systems.

CRAFT (Communication and Reasoning in Asymmetric Field Tasks) is a multi-agent coordination benchmark:

• Partial Information Environment: Each agent has a complementary but incomplete perspective
• Collaborative Goal: Coordinate via natural language to jointly build a 3D structure that no single agent can fully observe
• Formal Definition: Multi-sender practical reasoning task

Provides a systematic failure decomposition mechanism:

1. Spatial grounding errors
2. Belief modeling errors
3. Pragmatic communication errors

Includes a classification system for behavioral failure characteristics of cutting-edge models and open-source weight models.

Tests cover 8 open-source models and 7 cutting-edge models (including reasoning models):

• Reasoning ability ≠ Coordination ability: Stronger reasoning ability does not reliably translate into better coordination performance
• Small model counterattack: Smaller open-source models can often match or even outperform cutting-edge systems
• Individual ≠ Collective: Improving individual communication does not guarantee successful collaboration.

Multi-agent coordination remains a fundamental unsolved challenge for current language models.

https://github.com/csu-signal/CRAFT