MM-Fundus-CLIP: Research on a Multimodal Foundation Model for Fundus Images Integrating Large Language Models and CLIP

This project explores the development of a foundation model for fundus images using the CLIP contrastive learning architecture and large language models, enabling unified representation learning and cross-modal understanding of ophthalmic multimodal data. The project is maintained by myeongkyunkang and was published on GitHub (link: https://github.com/myeongkyunkang/mmfundusclip) in June 2026.

Fundus examination is a core method for diagnosing ophthalmic diseases, but the training cycle for professional ophthalmologists is long and their distribution is uneven. Deep learning brings hope for the automation of ophthalmic diagnosis, but traditional models are mostly designed for single tasks and have limited generalization capabilities. Foundation models learn general representations through pre-training on large-scale diverse data, providing new ideas to solve this problem. The MM-Fundus-CLIP project introduces the CLIP architecture and combines it with the semantic understanding capabilities of large language models to build a multimodal foundation model for fundus images and text.

CLIP is a multimodal learning framework proposed by OpenAI in 2021. Its core is to align image and text encoders in a shared embedding space through contrastive learning. Its advantage lies in zero-shot capability, but applying it to medical imaging faces challenges: medical images are highly professional, and ordinary text is difficult to capture pathological features; annotation costs are high, making it hard to obtain massive image-text pairs.

The project is based on the OpenCLIP framework and initialized with Apple's DFN5B-CLIP-ViT-H-14-384 model. The image encoder uses ViT-H/14 (384×384 resolution), the text encoder is a standard Transformer, and the output is projected into a 768-dimensional space. Training strategies include mixed-precision training (AMP BF16), gradient checkpointing, local loss calculation, etc. Hyperparameters: learning rate 1e-6, AdamW optimizer, batch size 128, 10 training epochs, 8 data loading processes.

Fundus images have a fixed structure (optic disc, blood vessels, macula, etc.), and pathological descriptions involve professional terms. The 'multimodal' aspect of MM-Fundus-CLIP may refer to image-text alignment, unified representation of different fundus image types, and modeling of different diseases. Potential applications include zero-shot disease screening, image-text retrieval, cross-modal retrieval, report generation, similar case retrieval, etc.

Advantages of foundation models: high data efficiency (adapting to downstream tasks with a small number of annotations), strong task generalization ability, good knowledge transferability, and interpretability supporting clinical applications. In terms of engineering: the code structure is clear (open_clip, open_clip_train directories), dependency management is explicit, and pre-trained models are hosted using Hugging Face Hub.

Challenges faced: data privacy compliance, the gap between general CLIP and the medical imaging field, clinical validation, and model interpretability. Future directions: develop a unified AI system, integrate multimodal data (fundus images, OCT, medical history, etc.), and achieve precise ophthalmic medical care.