Exploration of Large Language Models in Clinical Diagnosis Support: An Evaluation Study Based on MIMIC-IV

This project is a bachelor's degree study aimed at exploring the application of large language models (LLMs) in clinical diagnosis support and treatment recommendation systems. Using the real clinical database MIMIC-IV, the study tested the performance of various LLMs in symptom interpretation and diagnostic reasoning tasks through prompt engineering and accuracy evaluation, and assessed their practicality and limitations as diagnostic support tools.

Medical diagnosis requires integrating multiple aspects of information such as patient symptoms, medical history, and test results, placing a heavy cognitive load on doctors, especially in emergency departments or primary care institutions. The value of AI-assisted diagnosis systems is increasingly prominent. In recent years, LLMs have made breakthroughs in natural language tasks; some studies show their performance in medical exams is close to that of human medical students, but most use standardized questions, which have a gap with real clinical scenarios. The goal of this study is to explore the performance of LLMs on real clinical data and assess their practicality and limitations.

The study uses the MIMIC-IV (Medical Information Mart for Intensive Care) database, maintained by the MIT Laboratory for Computational Physiology, which contains real patient data from the Beth Israel Deaconess Medical Center (including hospitalization records, ICU monitoring data, laboratory test results, imaging reports, clinical notes, etc.). Advantages: 1. Authenticity (reflects actual medical complexity); 2. Diversity (covers various diseases, age groups, and conditions); 3. Richness (structured + unstructured data). Note: MIMIC-IV is a restricted resource that requires training and identity verification through PhysioNet to access; the code repository does not contain raw data, only processing scripts.

Data Extraction and Preprocessing

A specialized process was developed to extract patient information and clinical notes from the hosp and icu modules of MIMIC-IV, clean and structure them into LLM input format: extract demographic/diagnostic/medication history; remove sensitive information; convert unstructured notes into structured prompts.

Prompt Engineering

As a core technical link, multiple prompt templates were designed to test the impact of strategies. A good prompt needs to include patient background, symptom description, test results, and task instructions. Few-shot learning and chain-of-thought techniques were explored to guide the model in step-by-step reasoning.

Model Evaluation

Mainstream LLMs (GPT series, Llama, etc.) were evaluated using the following metrics: diagnostic accuracy (consistency with doctors' diagnoses), rationality of treatment recommendations (compliance with clinical guidelines), safety (no harmful recommendations), and interpretability (providing reasonable explanations).

MIMIC-IV Integration Module

Provides a data extraction pipeline, assuming users have obtained PhysioNet access and are running it locally.

Data Processing Scripts

Python scripts handle data cleaning, format conversion, and anonymization to ensure data integrity and compliance with privacy requirements.

Prompt Construction Tools

Flexible tools support generating customized prompt templates, enabling rapid experimentation with different strategies and result recording.

Evaluation Framework

Automatically evaluates model outputs, including comparison with standard diagnoses, an interface for clinical expert review, etc.

Clinical Potential of LLMs

It was confirmed that LLMs can extract key information from complex clinical notes and provide reasonable diagnostic recommendations, with the potential to become an auxiliary tool for doctors to reduce cognitive load.

Importance of Prompt Engineering

Prompt design significantly affects model performance; structured and clearly instructed prompts can produce more accurate and safe outputs, emphasizing the need to invest effort in optimizing prompts.

Limitations and Challenges

1. Knowledge timeliness: Training data has a cutoff date and may not include the latest treatment methods; 2. Hallucination problem: Generates seemingly reasonable but incorrect information; 3. Lack of clinical intuition: Cannot obtain information through physical examinations, etc.; 4. Responsibility attribution: The definition of responsibility when AI makes errors remains unresolved.

Human-Machine Collaboration Model

LLMs should serve as auxiliary tools rather than replace doctors, taking charge of information organization, preliminary screening, literature retrieval, etc., allowing doctors to focus on complex decision-making and patient communication.

Data Privacy and Security

Medical data is sensitive and requires strict compliance with privacy regulations. This study uses de-identified public datasets and emphasizes local processing, demonstrating responsible practices.

Continuous Evaluation and Monitoring

AI performance may change over time; a continuous evaluation and monitoring mechanism needs to be established, especially in high-risk medical fields where performance degradation may lead to serious consequences.

This project systematically explores the application of LLMs in clinical diagnosis support, providing empirical evidence using real MIMIC-IV data. LLMs cannot replace doctors currently, but their potential as auxiliary tools is significant. With technological progress and improved regulation, we look forward to more responsible medical AI applications entering clinical practice. For medical AI researchers, this project provides a reference for a complete research process (data acquisition, preprocessing, evaluation, analysis).