Multimodal AI for Predicting Pleural Invasion in Lung Cancer: A Fusion Practice of Deep Learning, Radiomics, and Clinical Biomarkers

This article introduces an open-source multimodal AI web application that integrates 3D deep learning, radiomic features, and clinical biomarkers to predict the risk of pleural invasion in lung cancer patients, providing an intelligent auxiliary tool for clinical decision-making. The project is maintained by Joeaicool, sourced from the GitHub project LungCancer-Pleural-AI, and was released on June 8, 2026.

Lung cancer is one of the malignant tumors with the highest incidence and mortality rates globally. Pleural invasion is a key indicator for evaluating lung cancer staging and prognosis, directly affecting the choice of treatment strategies. Traditional CT diagnosis relies on doctors' experience, which has problems such as strong subjectivity and poor consistency, so there is an urgent need for objective and accurate automated tools.

The project adopts a three-modal fusion strategy:

1. 3D Deep Learning Features: Extract 512-dimensional deep features using the MedicalNet pre-trained 3D ResNet;
2. Radiomic Features: Extract quantitative features such as original images, LoG transformation, wavelet transformation, and texture matrices via PyRadiomics;
3. Clinical Biomarkers: Incorporate three key indicators: CEA, CA125, and age.

• Classifier: Adopt Random Forest + Particle Swarm Optimization (PSO) for tuning; model weights are saved in RF_PSO_best.pkl;
• Deep Learning Architecture: Improve the ResNet-10 3D architecture, extract spatial features through pre-trained models;
• Preprocessing: Window level adjustment, ROI cropping, spatial resampling, and mask processing;
• Interpretability: Integrate the SHAP library to analyze feature contribution, enhancing decision trust.

Developed based on the Streamlit framework, the functions include:

• Data Upload: Support CT scans and tumor ROI masks (in .nii.gz format);
• Clinical Data Input: Entry of CEA, CA125, and age;
• Automatic Prediction: One-click execution of feature extraction and prediction, generating SHAP visualization;
• Result Display: Visualization of risk probability and feature importance ranking.

• Preoperative Evaluation: Objectively predict the risk of pleural invasion, assisting in the formulation of surgical plans;
• Auxiliary Decision-Making: SHAP explanations help doctors understand the basis of predictions, improving diagnostic consistency;
• Scientific Research and Teaching: The open-source framework promotes multimodal fusion research and can serve as a medical education tool.

Limitations: Small data scale, reliance on precise ROI annotations, single task; Future Directions: Integrate more modal data, develop automatic segmentation modules, conduct multi-center validation, and explore federated learning technology.