Innovative Application of Multimodal AI in Breast Cancer Screening: Integrated Diagnosis of Imaging and Clinical Data

This project innovatively integrates ultrasound imaging, clinical history, and molecular biomarkers to build a high-precision three-class breast cancer diagnosis system (benign, malignant, normal) using deep learning, providing a complete and reproducible technical solution for the implementation of medical imaging AI. The core lies in the effective integration of multimodal data and the design of a late-fusion architecture, balancing accuracy and engineering practicality.

Breast cancer is the most common malignant tumor among women worldwide, and early screening is crucial. Ultrasound examination is non-invasive and low-cost, but manual image reading has issues such as strong subjectivity, information silos, and heavy workload. Early single-modal AI only focuses on images, ignoring key information like age, medical history, and tumor markers, making multimodal fusion a key direction to improve diagnostic accuracy.

The project integrates three data modalities: ultrasound imaging (224×224 RGB images, providing morphological information), clinical history (25 features), and molecular biomarkers (10 laboratory indicators). A late-fusion strategy is adopted: ultrasound images are processed by an EfficientNet-B3 encoder to output 256-dimensional features; clinical/molecular data are processed by an MLP to output 64-dimensional features; after concatenation, the fused features are classified into three categories via a fusion head.

A two-stage training approach is used: in the Warmup phase, the EfficientNet backbone is frozen and only the classification head is trained; in the Fine-tune phase, end-to-end training is performed using cosine annealing learning rate and early stopping mechanism. To address class imbalance in the data (437 benign cases, 210 malignant cases, 133 normal cases), an inverse frequency weighting strategy is used to set class weights, ensuring the model focuses on minority classes.

The system includes a data preprocessing pipeline (image segmentation and enhancement; tabular feature encoding and standardization), training and evaluation scripts (supporting multimodal/single-modal training and baseline comparison), and an inference interface (single/batch prediction and multimodal association). It outputs class labels and probability distributions for easy clinical application.

The model is evaluated using multiple metrics such as macro-averaged F1, weighted F1, and ROC-AUC, which meet the needs of medical scenarios (high cost of missed malignant cases). Interpretability is enhanced through visualization of confusion matrices and ROC curves, providing decision-making references for doctors and reducing the missed diagnosis rate.

The project's highlights include centralized configuration management (unified hyperparameter configuration), modular code structure (separation of data/model/training), complete documentation (lowering the threshold for reproduction), and baseline comparison (learning gains between traditional machine learning and deep learning).

Currently, it can be applied to auxiliary diagnosis (second opinion), screening triage (primary screening at grassroots level), and teaching/training. Future expansion directions include introducing mammography/MRI modalities, adding time-series data, integrating PACS systems, and using federated learning to achieve multi-center data collaboration.