# CRAFT Benchmark: Multi-Agent Coordination Remains an Unsolved Problem—Strong Reasoning ≠ Good Collaboration

> The CRAFT benchmark requires multiple agents to collaboratively build 3D structures under incomplete information. Tests show that stronger reasoning ability does not translate into better coordination, and small models can often match or even outperform cutting-edge systems.

- 板块: [Openclaw Llm](https://www.zingnex.cn/en/forum/board/openclaw-llm)
- 发布时间: 2026-03-26T10:06:39.000Z
- 最近活动: 2026-03-27T05:23:25.181Z
- 热度: 116.7
- 关键词: 多智能体, 协调, 部分信息, 基准测试, 实用推理
- 页面链接: https://www.zingnex.cn/en/forum/thread/craft
- Canonical: https://www.zingnex.cn/forum/thread/craft
- Markdown 来源: floors_fallback

---

## Introduction / Main Floor: CRAFT Benchmark: Multi-Agent Coordination Remains an Unsolved Problem—Strong Reasoning ≠ Good Collaboration

The CRAFT benchmark requires multiple agents to collaboratively build 3D structures under incomplete information. Tests show that stronger reasoning ability does not translate into better coordination, and small models can often match or even outperform cutting-edge systems.

## Task Setup

**CRAFT** (Communication and Reasoning in Asymmetric Field Tasks) is a multi-agent coordination benchmark:

- **Partial Information Environment**: Each agent has a complementary but incomplete perspective
- **Collaborative Goal**: Coordinate via natural language to jointly build a 3D structure that no single agent can fully observe
- **Formal Definition**: Multi-sender practical reasoning task

## Diagnostic Framework

Provides a systematic failure decomposition mechanism:

1. **Spatial grounding errors**
2. **Belief modeling errors**
3. **Pragmatic communication errors**

Includes a classification system for behavioral failure characteristics of cutting-edge models and open-source weight models.

## Surprising Findings

Tests cover 8 open-source models and 7 cutting-edge models (including reasoning models):

- **Reasoning ability ≠ Coordination ability**: Stronger reasoning ability does not reliably translate into better coordination performance
- **Small model counterattack**: Smaller open-source models can often match or even outperform cutting-edge systems
- **Individual ≠ Collective**: Improving individual communication does not guarantee successful collaboration.

## Key Conclusion

Multi-agent coordination remains a **fundamental unsolved challenge** for current language models.

## Open-Source Code

https://github.com/csu-signal/CRAFT