Systematic Review of Large Language Models in Disease Diagnosis: Technical Pathways, Datasets, and Future Directions

This article is based on the latest review from Nature's sub-journal npj Artificial Intelligence 2025. It systematically organizes the technical routes, evaluation methods, public datasets, and future challenges of large language models (LLMs) in the field of disease diagnosis, providing a panoramic reference for medical AI researchers and practitioners. Completed by a team from institutions including the Hong Kong Polytechnic University, this review is the first to systematically sort out this emerging field and establish a structured analytical framework, helping to understand the applicable scenarios of different technical pathways, differences in evaluation methods, and considerations for dataset construction.

As large language models (LLMs) demonstrate strong understanding and reasoning capabilities in natural language processing tasks, their application potential in the healthcare field has attracted attention. Disease diagnosis, as a core medical link, involves complex clinical reasoning, multi-modal data fusion, and uncertain decision-making, making it an ideal scenario to test the capabilities of LLMs. The 2025 review not only summarizes major academic achievements but also establishes a structured analytical framework, providing a valuable introductory guide for developers in the medical AI field.

The review classifies existing research into four categories based on technical routes:

1. Retrieval-Augmented Generation (RAG)：Combines external medical knowledge bases to mitigate hallucinations, performs prominently in medical Q&A and diagnostic assistance, and can adapt to specific fields without expensive training;
2. Supervised Fine-tuning：Fine-tuned for specific diseases/specialties (e.g., OphGLM ophthalmic assistant, SkinGPT-4 dermatology system), where high-quality labeled data can significantly improve performance in specialized tasks;
3. Reinforcement Learning with Human Feedback (RLHF)：Aligns models with clinical experts' decision preferences (e.g., HuatuoGPT, Qilin-Med), suitable for complex differential diagnosis;
4. Specialized Medical Pre-trained Models：Requires large resource investment but achieves thorough capability improvement (e.g., ClinicalMamba, Biomistral), excels at handling longitudinal clinical records and cross-modal data.

Modern medical diagnosis relies on multi-modal data (imaging, physiological signals, laboratory tests, etc.). Leading research is exploring effective fusion:

• Visual-language models: In medical image interpretation (e.g., CXR-LLaVA for chest X-rays, PathGen for pathological images), they can identify lesions and generate standardized reports;
• Time-series data fusion: For example, ESI (ECG Semantic Integrator) realizes the conversion from signals like electrocardiograms to diagnoses;
• Challenges: Data alignment and model architecture design need to address issues such as sampling frequency differences across modalities, time alignment, and feature representation.

High-quality open datasets drive the progress of the field. The main datasets collated in the review include:

• MSDiagnosis (multiple sclerosis), OpenXDDx (open differential diagnosis), MedDX-Bench (medical diagnosis benchmark), DiagnosisArena (diagnostic capability evaluation), MedCaseReasoning (medical case reasoning), MedRBench (comprehensive medical reasoning benchmark), RareArena/RareBench (rare diseases), CUPCase (cancer of unknown primary), DDXPlus (extended differential diagnosis). These datasets provide standardized evaluation benchmarks, facilitate fair comparison of the effectiveness of different methods, and are valuable resources for developing medical AI applications.

Current evaluation methods in the field have discrepancies: different studies use varying metrics, test set divisions, and manual evaluation standards, making cross-study comparisons difficult. The main evaluation dimensions include diagnostic accuracy, rationality of differential diagnosis ranking, reasoning interpretability, performance compared to human doctors, etc. In the future, a more standardized evaluation framework needs to be established, especially for validity verification in real clinical scenarios—high performance in the laboratory does not equate to clinical practical value.

Limitations of existing research: Data privacy and ethical constraints hinder the construction of large-scale public datasets; model interpretability and clinical credibility need to be balanced (doctors need to understand the basis for decisions). Future directions: Develop efficient parameter fine-tuning techniques to reduce deployment costs; build cross-hospital and cross-population generalization verification mechanisms; explore best practices for human-machine collaboration; establish a regulatory and standard system for medical AI.

Large language models (LLMs) in the field of disease diagnosis are developing rapidly, with diverse technical routes and gradually enriched datasets, showing innovative vitality from laboratory to clinical verification. Technical developers need to understand the applicable boundaries of technical pathways, make good use of public data, and focus on real clinical needs. This review provides a clear roadmap for participants and is worth in-depth study.