Is Multi-Agent Always Better? A Controlled Variable Evaluation Study of LLM Agent Workflows

This study uses the BenchAgent standardized evaluation framework to challenge the common assumption of "more is better" through rigorous controlled variable experiments. The results show that only one out of six tested multi-agent systems is on par with the single-agent baseline, and most are inferior to single-agent in both accuracy and cost efficiency. The study provides evidence-driven design insights for the Agent field.

Currently, the LLM Agent field generally believes that increasing the number of agents can improve performance, but existing comparisons have methodological flaws (such as inconsistent benchmark loading, tool access, etc.). The core question of this study: Under standardized conditions, is multi-agent really better?

BenchAgent ensures consistency across all systems in dimensions such as benchmark loading, tool access, answer validation, cost calculation, and trajectory recording. The evaluation includes two dimensions: internal substrate (GPT-4.1 testing reasoning/coding/tool use) and external protocol alignment (GAIA benchmark testing dynamic workflows).

• SI Evaluation: Among the six multi-agent systems, only EvoAgent is on par with the single-agent; the remaining five are 2.56-11.29 percentage points behind, and have a worse cost-accuracy trade-off;
• PAE Evaluation: Dynamically generated workflows perform outstandingly on the GAIA benchmark, being more than 20 percentage points higher than the strongest fixed MAS.

1. Coordination Overhead: Extra costs such as inter-agent communication offset the benefits of division of labor;
2. Error Propagation: Errors cascade and amplify in chain/hierarchical architectures;
3. Rigid Predefined Architecture: Fixed role processes are not adapted to specific task requirements.

1. Single-Agent First: Optimize single-agent first, then consider multi-agent when encountering bottlenecks;
2. Dynamic Is Better Than Fixed: Dynamically generated workflows are more adapted to task requirements;
3. Strict Cost-Benefit Analysis: Consider accuracy, token consumption, latency, etc.

Limitations: Model (mainly GPT-4.1), task scope (not covering creative writing, etc.), limited MAS design space; Future Directions: Adaptive MAS, hybrid architecture, fine-grained task characteristic analysis, long-term interaction scenario research.