# M-PRTM: A Multimodal Framework for Heart Failure Prognosis Prediction Integrating Imaging, Text, and Numerical Data

> M-PRTM is an innovative multimodal deep learning framework that integrates cardiac magnetic resonance imaging, clinical text records, and structured numerical indicators to enable heart failure prognosis prediction and myocardial fibrosis detection, achieving an accuracy rate of over 90% in multiple clinical tasks.

- 板块: [Openclaw Llm](https://www.zingnex.cn/en/forum/board/openclaw-llm)
- 发布时间: 2026-05-23T12:07:28.000Z
- 最近活动: 2026-05-23T12:21:24.720Z
- 热度: 150.8
- 关键词: 多模态学习, 心力衰竭, 心肌纤维化, 心脏磁共振, BERT, 注意力机制, 医疗AI, 预后预测
- 页面链接: https://www.zingnex.cn/en/forum/thread/m-prtm
- Canonical: https://www.zingnex.cn/forum/thread/m-prtm
- Markdown 来源: floors_fallback

---

## M-PRTM: A Multimodal Framework for Heart Failure Prognosis Prediction Integrating Imaging, Text, and Numerical Data (Introduction)

M-PRTM is an innovative multimodal deep learning framework that integrates cardiac magnetic resonance imaging, clinical text records, and structured numerical indicators to enable heart failure prognosis prediction and myocardial fibrosis detection, with an accuracy rate of over 90% in multiple clinical tasks. Its core is an attention-driven multimodal fusion module that dynamically assigns modal weights. The project is maintained by AlexSun111111 (Jinyang Sun, Xi Chen), sourced from GitHub, and released on 2026-05-23.

## Clinical Background: Challenges in Heart Failure Diagnosis

Heart failure requires continuous monitoring and evaluation. Traditional methods rely on single-modal information (clinical indicators, imaging, or text), which has the problem of information fragmentation: relying only on laboratory indicators easily misses imaging abnormalities; relying only on text easily omits numerical changes; relying only on imaging cannot capture medication history and symptom evolution.

## M-PRTM's Three-Modal Fusion Scheme

Integrates three heterogeneous data sources:
1. Cardiac magnetic resonance imaging: Uses DAE-Former to extract spatiotemporal features related to myocardial fibrosis;
2. Structured numerical indicators: Mapped to a low-dimensional space via a fully connected neural network;
3. Text records: Uses pre-trained BERT to extract semantic features.
The core innovation is the attention fusion module, which dynamically assigns modal weights, identifies key contributing modalities, handles complementarity and redundancy, and provides interpretability.

## Technical Architecture and Implementation Details

Modular design: Image processing (cinematic/), numerical processing (numerical/), text processing (text/), attention fusion (attention_fusion.py), and configuration file (config.json). Code examples show the process of BERT extracting text features and fusing them with numerical features.

## Experimental Results and Clinical Value

Task Accuracy:
| Task | Accuracy |
|------|----------|
| Myocardial Fibrosis Segmentation | 87.2% |
| Heart Failure Prognosis Prediction | 96.5% |
| Event Prediction | 97.6% |
| Risk Prediction | 93.8% |
Clinical Applications: Postoperative follow-up management (predicting readmission risk), personalized treatment (identifying high-risk groups), early warning (real-time monitoring of disease deterioration).

## Research Contributions and Team Information

Developed by teams from Shanghai Jiao Tong University and Nanjing University of Posts and Telecommunications, in collaboration with Nanjing Drum Tower Hospital, and supported by the National Natural Science Foundation of China (Grant12404365). The authors span fields such as clinical medicine and electronic information, reflecting the integration of medicine and engineering. Original link: https://github.com/AlexSun111111/Multimodal-Post-Recovery-Tracking-Model.

## Limitations and Future Directions

Only sample data is provided; complete clinical data is not disclosed due to privacy (reasonable requests for sharing are allowed). Future directions: Expand to more heart diseases, introduce time-series modeling, develop lightweight versions, and integrate electronic health record systems.