Multimodal Data Fusion: An Early Detection System for Parkinson's Disease Based on Speech, Imaging, and Handwritten Spiral Analysis

This project constructs a multimodal early detection system for Parkinson's disease that integrates speech analysis, MRI image processing, and handwritten spiral image recognition. It uses explainable AI technologies (SHAP, LIME) to enhance model transparency, providing an innovative solution for the automated screening of neurodegenerative diseases.

Parkinson's Disease (PD) is the second most common neurodegenerative disease globally, with over 10 million patients worldwide and the number continues to grow. Typical symptoms (resting tremor, muscle stiffness, etc.) only appear in the middle and late stages; by the time of diagnosis, 60%-80% of dopaminergic neurons have been lost, missing the window for early intervention. Traditional diagnosis relies on clinical assessment, which is highly subjective and insensitive to early changes. Studies have found that PD affects speech, motor coordination, and brain structure, providing a theoretical basis for multimodal fusion diagnosis.

1. Speech Signal Analysis

Changes in speech features of PD patients: reduced volume, monotonous pitch, slurred pronunciation, and abnormal speech rate. The system extracts acoustic features such as Mel-Frequency Cepstral Coefficients (MFCC), fundamental frequency changes, jitter, and shimmer to train the model.

2. MRI Brain Image Analysis

MRI reveals atrophy in the substantia nigra region and increased iron deposition. The system uses deep learning to process 3D MRI data, automatically locate PD-related regions, and quantify structural abnormalities, which is more accurate than manual image reading.

###3. Handwritten Spiral Image Analysis When PD patients draw spirals, the lines have uneven thickness and irregular spacing. The system inputs the images into a Convolutional Neural Network (CNN) to identify movement disorder patterns.

The technology stack is based on the Python ecosystem:

• Data processing: Pandas, NumPy
• Speech analysis: Librosa
• Image processing: SimpleITK, NiBabel
• Deep learning: TensorFlow/Keras
• Explainable AI: SHAP, LIME

Applications of explainability:

• Speech model: Visualize the contribution of acoustic frequency bands
• MRI model: Generate saliency heatmaps of brain regions
• Fusion decision: Display weights and confidence levels of each modality

This enhances clinical trust and provides directions for model optimization.

Limitations of single modality: Speech is affected by environment/cold, MRI is costly, and handwriting depends on cooperation. A late fusion strategy is adopted: train single-modal models separately and then perform weighted fusion at the decision layer.

The fusion model dynamically adjusts weights: for example, when speech quality is poor, its weight is reduced, relying on imaging and handwriting results to enhance robustness in real scenarios.

• Early screening: Quickly identify high-risk groups in communities/physical examination centers
• Auxiliary diagnosis: Provide objective quantitative indicators to reduce subjective differences
• Efficacy monitoring: Track disease progression and treatment effects, supporting personalized plan adjustments
• Research tool: Standardized processes facilitate multi-center data integration

Limitations:

• Data scale: Training data comes from public datasets; sample size and diversity need to be expanded
• Clinical validation: Large-scale prospective cohort studies are needed to verify performance
• Real-time performance: Some models have long inference times and need optimization

Future directions:

• Introduce more modalities (gait analysis, wearable data)
• Develop lightweight edge deployment solutions
• Explore integration with electronic health record systems