New Discovery in Multi-Agent Collaboration: Behavioral Economics Games Predict AI Teams' Scientific Task Performance

Research shows that the collaborative characteristics of Large Language Models (LLMs) in behavioral economics games can reliably predict their performance in AI4Science multi-agent team tasks, providing a new tool for low-cost screening of collaborative models. This article will discuss in detail from aspects such as background, methodology, core findings, and practical significance.

LLM-based multi-agent systems show potential beyond single agents in scenarios like scientific discovery, code generation, and complex problem-solving, but success highly depends on effective coordination between agents. When there are shared resource constraints (e.g., GPU computing power, API call limits), the trade-off between cooperation and competition becomes critical—selfish agents may gain locally but harm the team's overall performance. Core question: How to predict an agent's multi-agent collaboration ability at the model selection stage?

The study constructs an evaluation system in two phases: 1. Behavioral economics game evaluation: Test 35 open-source LLMs in 6 classic games (Prisoner's Dilemma, Public Goods Game, Trust Game, etc.) to build a collaborative characteristic profile for each model; 2. AI4Science multi-agent tasks: Deploy models to real collaborative tasks such as data analysis, model building, and scientific report generation, evaluating three dimensions: accuracy, quality, and completion.

1. Collaborative characteristic profiles can robustly predict AI4Science task performance, even after controlling for factors like model size and basic capabilities; 2. Models with effective coordination (maintaining cooperation in repeated games), multiplicative investment (team investment with synergistic effects), and non-greedy strategies (prioritizing long-term team benefits) perform better; 3. Collaboration ability is a measurable attribute independent of general capabilities—strong models are not necessarily good collaborators.

Limitations of traditional evaluation methods: High cost of end-to-end testing, strong subjectivity in manual evaluation, and difficulty in generalization across tasks. Advantages of the game framework: Extremely low cost (few tokens per interaction), standardization (comparable and reproducible), strong generalization (abstracts the essence of collaboration), and interpretability (corresponds to specific cooperation mechanisms).

1. Team formation: Use game tests to screen models with matching collaborative characteristics; 2. Model fine-tuning: Enhance collaboration tendency through fine-tuning with game data; 3. Mechanism design: Design targeted incentives (e.g., set team rewards for selfish models, prevent free-riding for overly altruistic models).

Current limitations: 1. The mapping between games and specific tasks requires more fine-grained analysis; 2. Collaborative characteristics of closed-source models are not covered; 3. Static games are difficult to capture agents' dynamic adaptation strategies; 4. The Western economic background of the games may have cultural biases. Future research needs to delve into these directions.