UniBrain: A Unified Multimodal Model for Brain MRI Imputation and Medical Understanding

Addressing the common issue of modality missing in medical imaging, UniBrain proposes a unified multimodal large language model that can simultaneously perform brain MRI modality imputation and medical understanding. Through self-alignment strategy and dynamic hidden state mechanism, it achieves excellent performance in various disease diagnosis tasks. This study is from the arXiv platform, published on 2026-06-15, with the original title "Unified Multimodal Model for Brain MRI Imputation and Understanding", link: http://arxiv.org/abs/2606.16484v1.

Multimodal Large Language Models (MLLMs) show great potential in the medical field, but face two core challenges: first, the scarcity of high-quality training data (limited by privacy regulations, high annotation costs, and uneven data distribution); second, frequent data missing issues in clinical practice (patients cannot complete a full set of MRI scans, affecting the application effect of traditional multimodal models).

The core innovations of UniBrain include:

1. Unified Training Strategy: Jointly learning modality imputation and medical understanding capabilities, maintaining high diagnostic accuracy even when processing incomplete data;
2. Interleaved Data Flow Design: Autoregressive training, generating multimodal data and performing medical reasoning simultaneously;
3. Self-Alignment Strategy: Using dense image embeddings to capture fine-grained anatomical features without detailed image annotations;
4. Dynamic Hidden State Mechanism: Mitigating the exposure bias problem in long-context multimodal reasoning.

Experiments on multi-disease brain MRI datasets verified performance in three aspects:

1. Brain Image Imputation Capability: Generated images under high modality missing still maintain anatomical consistency and clinical usability;
2. Medical Understanding Capability: Cross-modal recognition of normal structures, detection of abnormal lesions, and association of lesions with clinical symptoms;
3. Disease Diagnosis Performance: Achieved excellent results in various brain disease diagnosis tasks even with incomplete modalities.

Technical Significance: Demonstrates the advantages of MLLMs in processing incomplete data in medical imaging; the unified training strategy provides new ideas for medical AI development; Application Prospects: Improves the diagnosis and treatment process in neurology/radiology (can still assist diagnosis without complete scans); the modality imputation function can be used for data augmentation to train other models.

Limitations: Only targets brain MRI data; generalization ability to other anatomical parts/imaging modalities (such as CT, ultrasound) needs to be verified; clinical safety and regulatory compliance of generated images require further research; Future Directions: Expand to more modalities/parts, integrate electronic medical record data, and develop efficient reasoning mechanisms to support real-time clinical applications.

UniBrain organically combines modality imputation and medical understanding through a unified training strategy, effectively addressing the challenge of clinical data missing, promoting the development of medical AI technology, and providing new possibilities for improving patient diagnosis and treatment experience and enhancing diagnosis efficiency.