Breast Cancer Detection System Based on Multimodal Deep Learning: An Intelligent Diagnostic Solution Integrating Imaging and Clinical Data

Original Author & Source:

• Original Author/Maintainer: iamMandana
• Source Platform: GitHub
• Original Title: Breast-Cancer-Detection-ML-Model
• Original Link: https://github.com/iamMandana/Breast-Cancer-Detection-ML-Model
• Source Publish/Update Time: 2026-05-28T22:15:14Z

Core Viewpoint: This article introduces a multi-input machine learning system that integrates mammography images and clinical tabular data. Trained on the CBIS-DDSM dataset, it demonstrates the application potential and technical implementation path of multimodal AI in medical image diagnosis.

Breast cancer is one of the most common malignant tumors among women globally. Early screening and diagnosis are crucial for improving patient survival rates. Traditional breast cancer screening mainly relies on radiologists' manual interpretation of mammography images, which is not only time-consuming and labor-intensive but also prone to being affected by subjective factors such as the physician's experience and fatigue levels.

In recent years, artificial intelligence technology has shown great potential in medical image analysis. However, AI models that rely solely on image data often fail to fully utilize patients' clinical background information, such as age, family history, and past medical history—information that is equally critical for diagnostic decisions. Therefore, developing multimodal AI systems that can handle both image data and clinical structured data has become an important direction for improving diagnostic accuracy.

The project builds a multi-input machine learning model, whose core innovation lies in its ability to process two heterogeneous data types in parallel:

Image Data Stream

The system receives mammography images as visual input and uses a deep learning convolutional neural network (CNN) to extract high-dimensional visual features. The CBIS-DDSM (Curated Breast Imaging Subset of DDSM) dataset serves as the training foundation. This dataset contains expert-annotated breast images covering normal, benign, and malignant lesions, providing reliable supervision signals for the model.

Clinical Data Stream

In addition to image information, the model also receives structured clinical tabular data input, which may include key clinical indicators such as patient age, breast density, symptom descriptions, and past screening results. This data is processed through fully connected layers to generate clinical feature vectors that complement the image features.

Multimodal Fusion Mechanism

The core of the project lies in designing an effective fusion strategy to integrate feature vectors from the image branch and clinical branch. Common fusion methods include early fusion (feature-level concatenation), mid-level fusion (intermediate layer interaction), or late fusion (decision-level weighting). Multimodal fusion allows the model to learn the correlation patterns between image features and clinical indicators—for example, the imaging characteristics of patients in specific age groups, or the impact of breast density on the difficulty of lesion identification.

CBIS-DDSM is a curated subset of the Digital Database for Screening Mammography (DDSM), which has undergone standardized organization and quality screening. The characteristics of this dataset include:

• Large-scale samples: Contains thousands of pathologically verified breast image cases
• Expert annotations: Each image is accompanied by regions of interest (ROI) and pathological results annotated by experienced radiologists
• Standardized format: Images are normalized in size and preprocessed, making them directly usable by machine learning models
• Balanced categories: Covers different pathological types such as normal, benign, and malignant, supporting classification and detection tasks

Training with such high-quality datasets helps the model learn clinically meaningful feature representations rather than merely fitting data noise.

From the project architecture perspective, the technical implementation of the system involves the following key links:

Data Preprocessing Pipeline: Medical images usually require preprocessing operations such as normalization, denoising, and enhancement to improve model training stability. Clinical tabular data needs to handle missing values, outliers, and perform standardized encoding.

Model Training Strategy: Considering the class imbalance in medical data (malignant cases are usually fewer than benign ones), techniques such as class weighting, oversampling, or focal loss may be used during training to ensure the model has sufficient sensitivity to the minority class (malignant lesions).

Validation and Evaluation: The evaluation of medical AI models requires strict cross-validation strategies. Common metrics include accuracy, sensitivity (recall), specificity, and AUC-ROC curves. Sensitivity is particularly critical because the cost of missing a malignant lesion is much higher than misdiagnosing a benign one.

Interpretability Design: AI systems deployed in clinical settings need to have a certain level of interpretability so that physicians can understand the basis of the model's decisions. This can be achieved through techniques such as Gradient-Weighted Class Activation Mapping (Grad-CAM) to visualize the image regions the model focuses on.

Potential application scenarios of the multimodal breast cancer detection system include:

Auxiliary Screening: Acting as a "second reader" to assist radiologists in detecting suspicious lesions that may have been overlooked, reducing the missed diagnosis rate.

Risk Stratification: Based on comprehensive analysis of images and clinical data, stratify patients' risks to guide the frequency and intensity of subsequent examinations.

Resource Optimization: In areas with limited medical resources, the AI system can serve as an initial screening tool, concentrating limited expert resources on high-risk cases.

Decision Support: Provide diagnostic references for less experienced physicians, shorten the learning curve, and improve diagnostic consistency.

Although multimodal AI shows promising prospects in breast cancer detection, practical deployment still faces several challenges:

Data Privacy and Compliance: Medical data involves sensitive personal information. Model training and deployment must strictly comply with regulations such as HIPAA and GDPR, and use technologies like federated learning and differential privacy to protect patient privacy.

Domain Generalization Ability: There are differences in imaging equipment, scanning parameters, and patient groups among different medical institutions. The model needs to have good cross-domain generalization ability or adapt to new environments through continuous learning.

Clinical Integration: The AI system needs to seamlessly integrate into existing clinical workflows and interface with infrastructure such as PACS (Picture Archiving and Communication Systems) and HIS (Hospital Information Systems).

Regulatory Approval: Medical AI products usually require approval from regulatory agencies such as the FDA and NMPA, which requires the model to have sufficient validation data and safety proof.

This project demonstrates a multimodal machine learning system that integrates breast imaging and clinical data, representing an important development direction of AI-assisted medical diagnosis. By effectively integrating heterogeneous data sources, the system is expected to improve the accuracy and efficiency of breast cancer screening and provide valuable support for clinical decision-making. With the continuous advancement of multimodal learning technology and the accumulation of medical data resources, similar intelligent diagnostic systems will be applied in more disease areas, promoting the realization of precision medicine.