Breast-Cancer-Multimodal-AI: A Benchmark Study of Multimodal Pathological Foundation Models in Breast Cancer Prognosis Prediction

This project builds a two-stage AI system for breast cancer that integrates pathological images, genomics, and clinical data for survival prediction and risk stratification. By comparing pathological visual encoders such as CONCH, UNI2, and CTransPath, the CONCH V+C+G cross-attention architecture achieved the best performance with a C-index of 0.609 on the TCGA-BRCA dataset, providing a benchmark reference for precise breast cancer prognosis.

Breast cancer is one of the most common malignant tumors among women globally. Early screening and precise prognosis are crucial for improving survival rates. While AI technology has made significant progress in medical imaging, single-modal data cannot fully reflect the complexity of the disease. To address this challenge, this project builds a multimodal AI system integrating pathological images, genomics, and clinical data for breast cancer survival prediction and risk stratification.

Stage 1: Population-Level Screening Task: Population-level breast cancer detection based on mammography; Dataset: VinDr-Mammo (5000 examinations, 20000 images), with support for CBIS-DDSM extension; Model: ConvNeXt-Base + 4-view attention fusion, achieving a test AUROC of 0.7407. Stage 2: Individual-Level Diagnosis and Prognosis Task: Integrate pathological images, gene expression, and clinical information for individualized risk assessment; Dataset: TCGA-BRCA (1054 pathological slides, 1094 RNA sequencing cases, 1097 clinical records); Evaluation endpoint: PFI (Progression-Free Interval).

Compared Encoders

• CONCH (Vision-Language): 512 dimensions, best performance in downstream survival prediction;
• UNI2 (Vision): 1536 dimensions, strong visual baseline;
• CTransPath (Vision): 768 dimensions, open-source and reproducible;
• Virchow (Vision): 1280 dimensions, candidate for future comparison. Optimal Architecture: CONCH V+C+G Cross-Attention Core metrics: C-index of 0.609±0.044; Stage1 AUROC of 0.741 (95% CI: 0.649-0.825); Log-rank test p-value of 0.005 for risk stratification; 5-year time-dependent AUC of 0.612.

Project Structure Covers the entire workflow from data preprocessing to deployment: agents/ (modules for mammography, visual encoders, etc.), training/ (multimodal fusion training), evaluation/ (evaluation and visualization), orchestrator/ (two-stage routing), apps/ (Streamlit demo), etc. Computing Infrastructure: UK Isambard-AI Supercomputing Center, NVIDIA GH200 GPU cluster (32 GPUs), supporting parallel feature extraction. Fusion Strategy: V+C+G cross-attention mechanism, which can independently encode features of each modality, learn inter-modal associations, dynamically balance contributions, and provide interpretable risk scores.

Precision Medicine Support: Risk stratification capability (Log-rank p=0.005) can identify high-risk patients and assist in formulating individualized treatment plans. Research Reproducibility: Adopts an open-source strategy, using public datasets and accessible foundation models, which is conducive to benchmark comparisons and method improvements in the field. Deployment Path: Streamlit demo supports rapid validation, SLURM scripts support supercomputing deployment, and two-stage routing logic supports integration into clinical workflows.

Current Limitations: Dataset is limited to TCGA-BRCA and requires multi-center validation; C-index of 0.609 still has room for improvement; No prospective clinical validation yet. Future Directions: Integrate more modalities (radiomics, proteomics); Explore advanced fusion architectures; Conduct prospective clinical trials; Develop an optimized version for real-time inference.

The Breast-Cancer-Multimodal-AI project demonstrates the potential of multimodal AI in precision oncology, providing empirical evidence for model selection in the field through systematic comparison of pathological foundation models. The open-source codebase and clear evaluation metrics lay the foundation for subsequent research, making it a valuable reference implementation for researchers and developers in AI medical applications.