Panoramic Applications of Large Language Models in Clinical Medicine: From Text Processing to Multimodal Diagnosis

This article systematically reviews the current applications of large language models (LLMs) in clinical medicine, covering 10 core task categories including text processing, medical literature mining, decision support, image/pathology report classification and generation, and multimodal diagnosis. It also organizes the development trajectory of 24 basic models and 42 medical-specific models. Additionally, it analyzes key challenges such as data privacy, hallucinations, clinical validation, and fairness, providing researchers and practitioners with references for technical roadmaps and open-source resource libraries.

With the rapid development of AI technology, LLMs have shown great potential in the medical field (e.g., electronic medical record processing, image interpretation, decision support, etc.), but related research is scattered and lacks systematic integration. To address this, Walid Mohamed et al. conducted a systematic review in accordance with the PRISMA 2020 standards, comprehensively analyzing 56 studies, establishing an open-source resource library, and dividing 10 core task categories under the framework of personalized medicine to provide a clear technical roadmap.

The 10 core applications of LLMs in clinical medicine include: 1. Clinical document processing and information extraction (e.g., GatorTron, NYUTron); 2. Medical literature mining (e.g., BioBERT, BioGPT); 3. Clinical decision support and question answering (e.g., Med-PaLM, ChatDoctor); 4. Radiology report classification (e.g., GPT-4, Mistral); 5. Pathology report classification (e.g., Path-llama3.1); 6. Radiology report generation (e.g., R2GenGPT); 7. Pathology report generation (e.g., WsiCaption); 8. Medical visual question answering (e.g., LLaVA-Med); 9. LLM-guided image segmentation (e.g., LLM4Seg); 10. Multimodal comprehensive diagnosis (e.g., GPT-4V).

Model evolution is divided into two parts: 1. Basic models (2017-2025): From the Transformer architecture to 24 models such as GPT, LLaMA, and Mistral, showing trends from pure text to multimodal, and from closed-source to open-source; 2. Medical-specific models (2020-2025): A total of 42 models, including three categories: text-specific (BioBERT, ClinicalBERT), biomedical (PubMedBERT), and multimodal (LLaVA-Med, Med-PaLM M).

Currently, LLMs face four major challenges in medical applications: 1. Data privacy and security: Need to comply with privacy regulations and explore technologies such as federated learning and differential privacy; 2. Hallucinations and reliability: Solved through methods like Retrieval-Augmented Generation (RAG) and human-in-the-loop validation; 3. Clinical validation and regulation: Need to undergo prospective, multi-center validation; 4. Fairness and bias: Need to address the issue of training data distribution bias.

LLMs are moving from the lab to the clinic, becoming intelligent partners for medical teams. This systematic review and resource library provide valuable references for practitioners, promoting the safe, effective, and fair application of LLMs. In the future, AI is expected to play an important role in improving medical quality, reducing costs, and promoting health equity.