LUMEN: A Large Language Model-Guided Multimodal Framework for Predicting Pulmonary Function from Chest CT

LUMEN is an LLM-guided multimodal medical AI framework whose core function is to predict pulmonary dysfunction from chest CT scan images. This framework innovatively integrates LLM semantic understanding with medical image analysis, addressing the limitations of traditional pulmonary function tests (e.g., difficulty in cooperation from critically ill/child patients), breaking through the shortcomings of single-modal deep learning, and providing a new tool for clinical diagnosis.

Early prediction of pulmonary dysfunction is crucial for the diagnosis and treatment of respiratory diseases. However, traditional tests require patients to cooperate with specific breathing movements, which is difficult for critically ill or child patients. Chest CT contains rich information about lung structures but it is hard to extract pulmonary function-related indicators. In recent years, deep learning has made significant progress in medical imaging but mostly focuses on single modalities. LUMEN introduces an LLM-guided mechanism to build a multimodal analysis framework for predicting pulmonary function indicators from CT images.

The core innovation of LUMEN lies in its "LLM-guided" design concept:

1. Multimodal Fusion Strategy: It includes a visual encoder (processing 3D CT images to extract deep features) and a language guidance module (using pre-trained LLMs to generate pulmonary function-related semantic descriptions and prior knowledge). Through an interaction mechanism, the fusion ensures that visual feature learning is constrained by medical semantics.

2. LLM Guidance Mechanism: The LLM acts as a "knowledge advisor", including prior knowledge injection (identifying pulmonary function-related anatomical structures in CT), feature alignment (contrastive learning to align visual and language features to enhance interpretability), and report generation (outputting predicted values and medical descriptions to assist doctors in understanding).

Key technical innovations of LUMEN:

1. Cross-modal Attention Mechanism: A new module allows multi-level interaction between visual and language features, focusing on pulmonary function-related regions in images and explaining them with medical terms.

2. 3D Image Processing Optimization: An efficient 3D convolutional network design ensures accuracy while controlling computational costs, making it deployable in clinical workflows.

3. Enhanced Interpretability: Through the LLM guidance mechanism, semantic explanations for prediction results are provided, allowing doctors to obtain numerical predictions and the medical basis for judgments.

LUMEN performs excellently in pulmonary function prediction tasks: compared with pure visual methods, LLM guidance significantly improves prediction accuracy and robustness; the model's interpretive descriptions are highly consistent with radiologists' evaluations.

Dataset and Evaluation Metrics: A large-scale chest CT dataset (covering various lung diseases and cases of different severity) was used. Evaluation metrics include traditional regression errors and clinical relevance indicators to ensure the prediction results have practical diagnostic significance.

Application Scenarios:

1. Early Screening: Integrate into routine CT workflows to automatically assess pulmonary function status, detect potential disorders, and enable early intervention;
2. Auxiliary Diagnosis: Provide quantitative indicators for patients with respiratory diseases to reduce the bias of doctors' subjective judgments;
3. Efficacy Evaluation: Compare CT scans at different times to objectively evaluate treatment effects and adjust plans.

Future Outlook: LUMEN demonstrates the potential of LLMs in medical AI, representing a new research paradigm of "LLMs as controllers", which can be extended to tasks such as tumor detection and organ segmentation. The open-source implementation provides a reference for researchers, and future multimodal technologies will improve the accuracy and efficiency of medical diagnosis.