OCTCube-M: 3D Multimodal OCT Foundation Model for Retinal and Systemic Diseases

OCTCube-M is a 3D optical coherence tomography (OCT)-based multimodal foundation model developed by the University of Washington team. It demonstrates exceptional disease prediction capabilities in cross-cohort, cross-device, and cross-modal validations. The model not only achieves state-of-the-art performance in retinal disease diagnosis but also predicts systemic diseases across organs. It has been open-sourced and opens up new avenues for AI-driven ophthalmic diagnosis.

Optical coherence tomography (OCT) is a core tool in modern ophthalmic diagnosis, enabling non-invasive acquisition of high-resolution 3D retinal images. However, extracting clinical information from complex 3D data is a key challenge for AI applications. The emergence of OCTCube-M addresses this issue. Developed by the University of Washington team and open-sourced on GitHub, it provides valuable resources for ophthalmic AI research and clinical applications.

OCTCube-M was pre-trained using over 26,685 3D OCT volume data (including 1.62 million 2D images). It adopts a Vision Transformer architecture combined with Flash Attention to improve efficiency. The project is implemented based on PyTorch 2.1.0 and CUDA 11.8, supporting Docker deployment to lower the barrier to use.

1. Retinal diseases: It achieves the best performance in predicting 8 common diseases (AMD, DME, POAG, DR, ERM, CRAO/CRVO, VD, RNV) and supports multi-task classification to improve diagnostic efficiency. 2. Cross-organ capabilities: It can predict systemic diseases such as lung nodule malignancy, reduced cardiac ejection fraction, diabetes, and hypertension. 3. Multimodal variants: OCTCube-IR enables OCT and infrared image retrieval, while OCTCube-EF integrates multimodal data to predict GA growth rate.

• Quick start: Download pre-trained weights and sample data, then perform inference via Jupyter Notebook. - Data preparation: Supports initialization with pre-trained models like RETFound and provides scripts for processing public datasets. - Open-source resources: Model weights are released on Hugging Face, including the original model, bimodal model, and multi-task classification model. Complete code is provided, and a saliency map generation tool is planned for release.

Clinical significance: Automated high-precision screening reduces the burden on doctors; cross-device compatibility facilitates commercial deployment; OCT may become part of whole-body health assessment. Future plans: Develop a trimodal OCTCube-EF model, OCTCube-IR inference code, and saliency map generation code to enhance practicality and interpretability.

OCTCube-M represents the latest progress in medical imaging AI foundation models in the ophthalmology field. Through large-scale pre-training, innovative multimodal architecture, and rigorous cross-domain validation, it pushes the boundaries of retinal disease diagnosis, demonstrates the great potential of foundation models in medical imaging, and is expected to become an important infrastructure for AI-driven ophthalmic diagnosis.