Mammo-CLIP: A Vision-Language Foundation Model Empowers Breast Imaging Analysis, Securing Top 11% at MICCAI 2024

Mammo-CLIP is the first vision-language foundation model specifically designed for breast imaging. By integrating breast imaging data with radiology report text, it achieves breakthroughs in data efficiency and model robustness, and was successfully selected as an outstanding paper in the top 11% at MICCAI 2024, providing an efficient AI-assisted tool for early screening and accurate diagnosis of breast cancer.

Breast cancer is one of the most common malignant tumors among women worldwide. Early screening and accurate diagnosis are crucial for improving survival rates. Mammography (X-ray) is the main screening method, but its interpretation relies on physician experience and has subjective differences. Traditional deep learning models face three major challenges: difficulty in obtaining labeled data, insufficient generalization ability, and lack of interpretability in decision-making processes. Mammo-CLIP addresses these pain points specifically.

Mammo-CLIP draws on the core ideas of CLIP and customizes it: it includes a visual encoder (for processing breast images) and a text encoder (for processing radiology reports), and achieves image-text alignment through contrastive learning. Data preprocessing supports DICOM-to-PNG conversion and handles multi-view images; pre-training uses a two-stage strategy: first, general image-text pairing training, then fine-tuning with breast image-report data to learn fine-grained correspondence between signs and text.

In downstream task evaluation, Mammo-CLIP performs excellently: its few-shot performance in benign/malignant breast classification is close to traditional methods; BI-RADS classification prediction is close to the level of senior physicians; lesion detection and localization enhance interpretability. Core advantages: excellent data efficiency (only hundreds of labeled samples needed), and strong cross-dataset generalization ability (performance degradation is less than traditional models).

Mammo-CLIP open-sources a complete toolchain: pre-trained weights are uploaded to Hugging Face, and data preprocessing, training, evaluation scripts, and tutorials are provided. Recently, it integrated LADDER (an ACL 2025 accepted paper), which can automatically identify model biases in subpopulations (such as dense breasts, lesions near implants) and generate correction strategies to improve system fairness.

Limitations of Mammo-CLIP: pre-training data mainly consists of English reports, and it does not cover 3D modalities (such as DBT). Future directions include integrating multi-modal information like ultrasound/MRI, developing multi-language versions, and deeply integrating with clinical decision systems. This model represents an important direction of medical imaging AI from single-modal to multi-modal fusion, and is expected to become a powerful assistant for radiologists.