ClinicRealm Study: Re-evaluation of Large Language Models in Clinical Prediction Tasks

ClinicRealm, a large-scale benchmark study published in npj Digital Medicine, systematically compared the performance of 15 GPT-style LLMs, 5 BERT models, and 11 traditional methods on non-generative clinical prediction tasks. It reveals that modern LLMs can outperform traditionally fine-tuned models in zero-shot settings, and leading open-source LLMs can match or even exceed proprietary commercial models, providing new evidence for medical AI selection.

Large Language Models (LLMs) are increasingly used in the medical field, but their utility in non-generative clinical prediction tasks has long been considered inferior to specially trained traditional machine learning models, leading to disputes and misuse risks in the medical AI field. The core issue lies in the lack of systematic benchmark tests to objectively evaluate the real capabilities of LLMs. The traditional view holds that encoder models like BERT are more suitable for structured EHR data after fine-tuning, while GPT-style models are better at text generation. However, it is worth re-examining whether new-generation large models (such as GPT-4, DeepSeek-V3) break this boundary.

The study built the ClinicRealm benchmark platform to evaluate three categories of models:

1. GPT-style LLMs (15)：Including GPT-4, GPT-5, DeepSeek-V3, DeepSeek-V3.1-Think, Claude series, etc.
2. BERT-style encoder models (5)：Including medical pre-trained models like ClinicalBERT, BioBERT, etc.
3. Traditional machine learning methods (11)：Including logistic regression, random forest, XGBoost, neural networks, etc.

The evaluation covers two types of data:

• Unstructured clinical text：Medical records, discharge summaries, consultation opinions, etc.
• Structured EHR data：Laboratory results, vital signs, diagnosis codes, etc.

Clinical Text Prediction: Breakthrough Performance of LLMs

Leading LLMs outperform fine-tuned BERT models in zero-shot settings. DeepSeek-V3.1-Think and GPT-5 perform the best. Zero-shot prompts can achieve or exceed the effects of traditional supervised learning, allowing high-performance models to be deployed without a large amount of labeled data.

Structured EHR Data: Trade-off in Data Efficiency

When data is sufficient, traditional models (such as XGBoost) are optimal; in data-scarce scenarios, the zero-shot capabilities of LLMs like GPT-5 and DeepSeek-V3.1-Think often exceed traditional methods, which is of significant value for scenarios like rare disease prediction.

Rise of Open-Source Models

Leading open-source LLMs (such as DeepSeek-V3.1-Think) can match or even exceed proprietary models, providing cost-effective, customizable, and auditable solutions for medical institutions, reducing reliance on commercial APIs, and enhancing data privacy protection.

ClinicRealm also evaluates the reasoning ability, reliability, and fairness of models: Reasoning ability: Whether LLMs can provide explainable prediction bases, the impact of chain-of-thought prompts on prediction quality, and the accuracy of internal medical knowledge representation. Reliability: Consistency of performance across different patient groups, robustness to input perturbations, and the degree of calibration of prediction confidence. Fairness: Performance differences across different race/gender/age groups, and potential bias amplification issues.

Model Selection Strategy

1. Prioritize LLMs for text tasks;
2. Consider zero-shot LLMs for data-scarce scenarios;
3. Need to balance structured data tasks (traditional methods still have advantages when data is sufficient and interpretability requirements are high).

Deployment Cost Calculation

Need to comprehensively consider: data annotation costs (zero-shot saves labor), model maintenance costs (a single general model replaces multi-task models), and development iteration speed (prompt engineering is faster than training).

Limitations:

• The evaluation focuses on prediction tasks and does not cover all clinical AI scenarios;
• Partially based on public datasets, which may differ from real clinical environments;
• Long-term safety and ethical impacts need continuous monitoring.

Future Directions:

• Expand to more tasks (such as drug recommendation, treatment plan generation);
• Explore hybrid architectures of LLMs and specialized models;
• Develop model compression and inference optimization techniques for medical scenarios.

The ClinicRealm study provides important empirical evidence for medical AI: modern LLMs are no longer just text generation tools, but have become strong competitors in clinical prediction, challenging traditional cognition and providing new ideas for AI technology selection. The rapid progress of open-source models is expected to promote a more open, efficient, and fair medical AI ecosystem.