HybridMFP: A Multimodal Fusion Deep Learning Model for Accurate Assessment of Upper Limb Motor Function Post-Stroke

HybridMFP is a hierarchical multimodal fusion deep learning model that integrates surface electromyography (sEMG) and kinematic signals (KIN) to achieve automated and objective prediction of FMA-UE scores for upper limb motor function post-stroke. It addresses limitations of traditional assessments such as subjectivity and time-consuming processes, providing technical support for intelligent rehabilitation.

Stroke is the leading cause of disability in adults worldwide, with 15 million new cases each year and one-third of patients left with permanent disabilities, among which upper limb impairment is common. Clinical assessment using FMA-UE has issues like reliance on subjectivity, time consumption, and inability to capture subtle dynamic changes. The urgent need for precise and intelligent assessment led to the development of the HybridMFP project.

Data Collection and Preprocessing

• sEMG: 12 channels covering upper limb muscle groups; KIN: 63-dimensional kinematic features (e.g., joint angles)
• Preprocessing: DTW time-series alignment, Z-score normalization to eliminate individual differences

Hierarchical Fusion Network

• Intra-modal learning: Convolution + time-series modeling to extract specific features
• Early fusion: Cross-attention/concatenation to learn inter-modal correlations
• High-level fusion: Fully connected layers to map to FMA-UE scores

Features and Model

• Elastic net for feature selection, PCA for dimensionality reduction
• Ensemble learning regression to predict scores

Leave-one-subject cross-validation was used, achieving an MAE of 0.70 points and an accuracy of 85%, with a 10% reduction in error and a 27% increase in accuracy compared to the baseline. Multimodal fusion outperforms single-modal and traditional fusion methods; the model has robust generalization and is highly correlated with expert assessments.

Breakthrough in Objective Assessment

Eliminate assessment discrepancies, enable high-frequency monitoring, and support tele-rehabilitation

Intelligent Rehabilitation Closed Loop

Function stratification matching schemes, intelligent training planning, prognosis assessment, real-time efficacy monitoring

Lightweight Accessibility

Wearable sEMG + low-cost motion capture, facilitating promotion in primary care and home settings

Current Limitations

Limited sample size, sEMG affected by electrodes/skin, only for standardized movements

Future Directions

Multicenter validation, real-time assessment optimization, multi-task learning, causal inference, rehabilitation digital twin

Open datasets:

• Kaggle1: https://www.kaggle.com/datasets/jinanzuo/kin-fma
• Kaggle2: https://www.kaggle.com/datasets/wuxingang/rocky-data These include synchronized sEMG, KIN signals, and FMA-UE scores to facilitate research reproduction.