Multimodal Skin Cancer Detection: When Medical Imaging Meets Patient Data

The MADS project team at the University of Michigan explores multimodal machine learning models combining medical imaging and patient metadata. They compare performance differences between single-modal and fusion schemes on the Stanford MRA-MIDAS dataset, aiming to provide more reliable AI-assisted tools for clinical diagnosis.

Skin cancer is one of the most common malignant tumors globally, and early detection is crucial for treatment outcomes. Traditional diagnosis relies on dermatologists' visual observation and empirical judgment, while artificial intelligence intervention has brought new possibilities for large-scale screening. However, deep learning models relying solely on medical images often ignore patient background information—metadata like age, gender, and medical history actually contain important diagnostic clues.

The capstone project of the University of Michigan's Master of Applied Data Science (MADS) program focuses on the Stanford MRA-MIDAS skin cancer dataset, a valuable resource combining high-quality dermoscopic images with rich patient metadata. MRA-MIDAS stands for 'Medical Record Analysis for Melanoma Detection using Image Analysis and Structured data', with its core goal to explore more effective fusion methods for visual information and structured data. The project compares three modeling strategies: image-only convolutional neural networks, metadata-only tabular models, and multimodal architectures fusing both, to quantify the independent contributions and synergistic effects of each information source.

The image processing branch uses a pre-trained deep learning backbone to extract visual features of skin lesions (color distribution, texture patterns, boundary irregularities, etc.); the metadata branch processes demographic features and clinical history. The multimodal fusion layer explores three strategies: early fusion (feature-level concatenation), mid fusion (joint representation after separate encoding), and late fusion (weighted integration after independent prediction). Each strategy has different trade-offs: early fusion is efficient but may cause modal conflicts, while late fusion preserves modal specificity but easily misses cross-modal interactions.

The project focuses on model uncertainty estimation. In medical scenarios, 'knowing what you don’t know' is more important than making wrong high-confidence predictions. Through ensemble methods or Bayesian neural networks, the model outputs a confidence score for each prediction, helping doctors identify difficult cases requiring manual review. This capability addresses issues like image quality differences and out-of-distribution samples, prevents overconfident misdiagnosis, and is crucial for practical deployment.

Through feature importance analysis and ablation experiments, the project identifies key factors affecting classification results (e.g., certain lesions are more common in specific age groups/skin tone populations), guiding model attention allocation. Interpretability meets the transparency requirements of medical AI and provides insights for clinical decision-making: doctors not only know the results but also understand the reasoning logic (based on image patterns or combinations of patient risk factors).

Multimodal detection represents the direction of precision medicine. Integrating multi-source data to obtain a comprehensive patient profile enhances primary care doctors' diagnostic ability in grassroots settings, reducing missed diagnoses and misdiagnoses. Future directions include expanding lesion types, integrating genomic data, developing real-time diagnostic mobile applications, and combining wearable devices and telemedicine to achieve dynamic risk assessment.