Multimodal Heart Disease Detection: An Intelligent Diagnostic System Fusing ECG Images and Clinical Data

This project was developed by AbdullahZahid1, with source code available on GitHub (link: https://github.com/AbdullahZahid1/Heart-Disease-Detection-using-Multimodal), released on June 8, 2026. Its core is to fuse electrocardiogram (ECG) images with clinical data, build an intelligent diagnostic system using CNN and XGBoost, and realize real-time risk classification through a Streamlit application, demonstrating the practical application value of multimodal machine learning in medical diagnosis.

Heart disease is a major global health threat, and early accurate diagnosis is crucial for prognosis. Traditional diagnosis relies on doctors' experience and requires integrating multimodal information such as ECG, blood indicators, and medical history, but a single data source is difficult to provide complete evidence. ECG images contain temporal features of cardiac electrical activity, while clinical data provides demographic and biochemical indicators. How to effectively fuse these two heterogeneous data types to build a more accurate model is an important research direction in the field of medical AI.

The project proposes an innovative multimodal fusion method that combines deep learning and traditional machine learning:

1. ECG Image Analysis Module: Uses CNN to process ECG images and extract waveform features and abnormal patterns;
2. Clinical Data Analysis Module: Uses XGBoost to process structured clinical data and capture nonlinear relationships between features;
3. Multimodal Fusion Layer: Fuses ECG features extracted by CNN with clinical features learned by XGBoost to form a comprehensive diagnostic basis, which is more robust than a single modality.

CNN Processing of ECG Images

ECG images contain key waveforms such as P waves, QRS complexes, and T waves. CNN extracts features from low-level edges to high-level semantic features through multiple layers of convolution and pooling, and may use pre-trained models (such as ResNet, VGG) for transfer learning or lightweight custom architectures.

XGBoost Modeling of Clinical Data

Clinical data includes risk factors such as age, gender, blood pressure, cholesterol, and blood glucose. XGBoost handles feature interaction effects by integrating multiple decision trees and can also evaluate feature importance to enhance interpretability.

Feature Fusion Strategy

Early fusion (feature-level concatenation), late fusion (decision-level weighting), or hybrid strategies may be adopted. The fused features are input into a classifier to output risk prediction results.

The project provides a Streamlit-based web application with the following advantages:

• Rapid Prototype Development: Builds the interface with pure Python code, no front-end experience required;
• Real-Time Inference Demo: Users upload ECG images + input clinical indicators, and the system returns prediction results and risk classification in real time;
• Visualization Display: Intuitively presents prediction results, feature importance, confidence, etc., to help understand AI decisions;
• Easy Deployment: Can be easily deployed to the cloud or locally, facilitating clinical verification and demonstration.

Compared with single-modal diagnosis, the fusion scheme of this project has three major advantages:

1. Enhanced Complementarity: ECG provides electrophysiological information, while clinical data provides metabolic and demographic information; their complementarity improves the comprehensiveness of diagnosis;
2. Improved Robustness: When the quality of one modality's data is poor or missing, the other modality can still provide a basis, enhancing the system's fault tolerance;
3. Performance Optimization: Proper fusion is usually better than the best performance of a single modality, achieving the effect of 1+1>2.

Application Prospects

• Auxiliary Diagnostic Tool: Provides a second opinion for doctors, especially suitable for primary care scenarios where specialist doctors are lacking;
• Early Screening System: Integrated into physical examination processes to quickly identify high-risk groups;
• Telemedicine Support: Combined with mobile devices and cloud computing to provide remote heart assessment for remote areas;
• Health Monitoring Integration: Can be combined with wearable devices in the future to realize continuous cardiovascular monitoring and early warning.

Technical Insights

References for developers:

• Preprocessing and feature extraction methods for different modal data;
• Effective combination strategies of CNN and gradient boosting models;
• Practice of using Streamlit to quickly build AI application prototypes;
• Data privacy and ethical considerations in medical AI development.

This project is a complete example of a multimodal medical AI application. By fusing ECG images with clinical data and combining the advantages of CNN and XGBoost, it realizes intelligent prediction of heart disease risk. The supporting Streamlit application allows rapid demonstration and verification of technical results, providing valuable references for the implementation of medical AI.