M-PRTM: A Multimodal Framework for Heart Failure Prognosis Prediction Integrating Imaging, Text, and Numerical Data

M-PRTM is an innovative multimodal deep learning framework that integrates cardiac magnetic resonance imaging, clinical text records, and structured numerical indicators to enable heart failure prognosis prediction and myocardial fibrosis detection, with an accuracy rate of over 90% in multiple clinical tasks. Its core is an attention-driven multimodal fusion module that dynamically assigns modal weights. The project is maintained by AlexSun111111 (Jinyang Sun, Xi Chen), sourced from GitHub, and released on 2026-05-23.

Heart failure requires continuous monitoring and evaluation. Traditional methods rely on single-modal information (clinical indicators, imaging, or text), which has the problem of information fragmentation: relying only on laboratory indicators easily misses imaging abnormalities; relying only on text easily omits numerical changes; relying only on imaging cannot capture medication history and symptom evolution.

Integrates three heterogeneous data sources:

1. Cardiac magnetic resonance imaging: Uses DAE-Former to extract spatiotemporal features related to myocardial fibrosis;
2. Structured numerical indicators: Mapped to a low-dimensional space via a fully connected neural network;
3. Text records: Uses pre-trained BERT to extract semantic features. The core innovation is the attention fusion module, which dynamically assigns modal weights, identifies key contributing modalities, handles complementarity and redundancy, and provides interpretability.

Modular design: Image processing (cinematic/), numerical processing (numerical/), text processing (text/), attention fusion (attention_fusion.py), and configuration file (config.json). Code examples show the process of BERT extracting text features and fusing them with numerical features.

Task Accuracy:

Task	Accuracy
Myocardial Fibrosis Segmentation	87.2%
Heart Failure Prognosis Prediction	96.5%
Event Prediction	97.6%
Risk Prediction	93.8%
Clinical Applications: Postoperative follow-up management (predicting readmission risk), personalized treatment (identifying high-risk groups), early warning (real-time monitoring of disease deterioration).

Developed by teams from Shanghai Jiao Tong University and Nanjing University of Posts and Telecommunications, in collaboration with Nanjing Drum Tower Hospital, and supported by the National Natural Science Foundation of China (Grant12404365). The authors span fields such as clinical medicine and electronic information, reflecting the integration of medicine and engineering. Original link: https://github.com/AlexSun111111/Multimodal-Post-Recovery-Tracking-Model.

Only sample data is provided; complete clinical data is not disclosed due to privacy (reasonable requests for sharing are allowed). Future directions: Expand to more heart diseases, introduce time-series modeling, develop lightweight versions, and integrate electronic health record systems.