MedCTA: A New Benchmark for Evaluating Clinical Tool Agents, Revealing Vulnerabilities of Multimodal Medical AI

MedCTA is a clinical tool agent evaluation benchmark released by the KAUST team, designed to test the performance of multimodal models in real clinical tasks. This benchmark includes 107 real clinical tasks and tested 18 multimodal models. The results reveal that cutting-edge models have vulnerabilities in multi-step clinical tool usage, such as protocol failures, premature termination, and incorrect tool calls.

Source Information:

• Team: KAUST Research Team
• Release Platform: arXiv
• Release Date: June 10, 2026
• Project Homepage: https://ivul-kaust.github.io/MedCTA/
• Original Paper Link: http://arxiv.org/abs/2606.11702v1

Medical AI is developing rapidly, but existing systems mostly stay at the level of simple image recognition or single-turn question answering, which cannot meet the complex capabilities required for real clinical decision-making, such as tool retrieval, evidence acquisition, and multi-source information integration.

Current evaluation benchmarks only focus on isolated perception tasks or single-turn QA, which cannot reveal the failures of agents in planning, tool recruitment, and rollout reliability, easily creating the illusion that models are competent for real clinical work. MedCTA was created precisely to fill this evaluation gap.

Core design features of the MedCTA benchmark:

1. Real Multimodal Input: Built based on real clinical data such as CT, MRI, pathological slices, and clinical reports;
2. 107 Real Tasks: Each task includes a doctor-validated trajectory, a sequence of 5 tool operations, and implicit goals for each step;
3. Process-Aware Evaluation Framework: Fine-grained evaluation from 5 dimensions—tool selection, parameter validity, execution stability, trajectory fidelity, and result quality—to accurately identify failure modes.

Test results on 18 multimodal models show:

• Cutting-edge models are still vulnerable: Systemic issues such as protocol failures (skipping/incorrect steps), premature termination, and incorrect tool recruitment exist;
• Perception ≠ Agent capability: Excellent image/text perception capabilities cannot automatically translate into reliable clinical agent behavior;
• Limitations of golden-standard routing: Even if humans specify the tool routing, the model's performance improvement is limited, with problems involving multiple links such as parameter generation and context integration.

Implications of MedCTA results for the development of medical AI:

1. Evaluation paradigm innovation: Need to focus on end-to-end task completion capabilities rather than isolated metrics;
2. Architecture redesign: Need to enhance planning modules, error recovery mechanisms, and reliable parameter generation;
3. Clinical validation first: All tasks should be validated by clinicians to ensure alignment with real needs.

MedCTA has made the following resources public:

• 107 clinical tasks and validated trajectories;
• Interface definitions for 5 deployed tools;
• Complete evaluation code and metric implementations;
• Detailed results of 18 tested models.

Openness helps researchers audit models, diagnose failure modes, and track progress.

MedCTA is not only an evaluation benchmark but also a sober examination of the current state of medical AI, revealing the distance to reliable clinical agents. When pursuing model scale and performance, we need to pay attention to reliability and safety.

MedCTA provides a strict testing platform for developing trustworthy clinical AI agents, and it is a must-read resource and essential tool for relevant researchers and engineers.