MedSP1000: Dynamic Evaluation of LLM Clinical Decision-Making Reveals 60% Accuracy Ceiling

The MedSP1000 standardized patient benchmark test shows that the state-of-the-art GPT-5.5 only completes 60.4% of expert-scored items in clinical decision-making tasks, while medical-specific models only reach 40%, and increasing reasoning computation does not lead to significant improvement. This dynamic evaluation exposes the core flaws of current LLMs in clinical scenarios, suggesting that they are not yet suitable for direct clinical deployment.

Large language models have broad application prospects in the medical field, but static single-round benchmark tests cannot truly reflect their performance in clinical scenarios. Real clinical decision-making is a dynamic process: it requires continuous information collection, adjustment of diagnostic hypotheses, and revision of treatment plans. Traditional question-and-answer tests ignore key dynamic interactions and process quality.

Standardized Patient Method

Drawing on the standardized patient (SP) model in medical education, the first interactive clinical agent benchmark test was created.

Dataset Scale

Includes 1638 cases, 24602 trajectory-level scoring criteria, complete case scripts, and clinical environment context.

Evaluation Framework

• Closed-loop interaction simulation: clinical agent (model under test), patient agent (standardized script), environment controller (process management)
• Process-level scoring: covers information collection quality, diagnostic reasoning process, appropriateness of treatment decisions, and patient communication skills

Model Performance Comparison

Model Type	Representative Model	Completion Rate of Scored Items
General-purpose LLM (Optimal)	GPT-5.5	60.4%
Medical-specific Model	Med-PaLM, etc.	40.0%
Other General-purpose Models	Llama3, Qwen, etc.	30-50%

Key Findings

1. Obvious performance ceiling: GPT-5.5 still has 40% clinically relevant flaws
2. Medical-specific models lag behind: deviation between training data and clinical scenarios
3. Ineffective reasoning computation: increasing resources does not improve performance

Failure Modes

• Information collection flaws: jumping to conclusions too early, missing key symptoms
• Reasoning issues: incomplete differential diagnosis, confirmation bias
• Treatment errors: inappropriate plans, dosage mistakes, ignoring contraindications

The study clearly points out that the defect rate of current LLMs (including medically fine-tuned models) reaches 40-60%, meaning that every 2-3 patients may receive improper diagnosis and treatment, and the risk of missed diagnosis and misdiagnosis is unacceptable. Evaluation methods need to shift from result-oriented to process-oriented, static to dynamic, and single-dimensional to comprehensive.

Future Research Directions

• Multimodal fusion: integrate multi-source information such as images and laboratory tests
• Long-term follow-up simulation: evaluate chronic disease management capabilities
• Team collaboration scenarios: simulate multidisciplinary consultations
• Enhanced interpretability: improve the transparency of reasoning processes

Implications

• Practitioners: need to optimize evaluation methods, use clinical-relevant training data, and enhance reasoning capabilities
• Public: human clinical judgment is still irreplaceable; caution is needed before AI matures