AgentRx: A Benchmark Study of LLM Agents in Multimodal Clinical Prediction Tasks

This study conducts a systematic benchmark evaluation of LLM agents in multimodal clinical prediction tasks. The core finding is that single-agent frameworks outperform naive multi-agent systems in aspects such as multimodal data processing and prediction calibration, providing a new evaluation benchmark for the medical AI field, and open-sourcing relevant code and frameworks to support community development.

Building effective clinical decision support systems requires integrating heterogeneous multimodal data (e.g., electronic health records, medical images, clinical notes, etc.), but most LLM agent research focuses on text modalities. The fragmentation of medical data makes multi-agent collaboration a potential solution, but the effectiveness of LLM agents in multimodal clinical prediction has not been fully verified.

AgentRx is a benchmark study of LLM agents for multimodal clinical prediction tasks. The team uses large-scale real-world data to systematically evaluate the performance of LLM agents, compare performance differences between unimodal and multimodal settings, and quantify the performance gap between single-agent and multi-agent systems.

1. Single-agent advantage: Performance is better than naive multi-agent systems, challenging the intuitive assumption that "multi-agent is necessarily better"; 2. Multimodal processing capability: More effectively integrates heterogeneous information from different sources; 3. Calibration performance: Better prediction calibration, which directly affects doctors' trust and willingness to adopt the system.

Current multi-agent systems have three shortcomings: the information fusion mechanism lacks effective cross-agent integration strategies, task allocation is not fully optimized, and communication overhead may introduce delays and error propagation.

The research team open-sourced the code and evaluation framework, providing valuable resources for the medical AI community, supporting fair comparison of the performance of different agent architectures, identifying best practices for multimodal fusion, and promoting the practical deployment of clinical AI systems.

In practical deployment, single-agent design is more cost-effective. Future research needs to focus on developing intelligent multi-agent collaboration protocols, exploring hybrid architectures that combine single-agent efficiency with multi-agent specialization, establishing agent evaluation standards for medical scenarios, and emphasizing the importance of sufficient benchmark testing before deployment.