Multimodal Depression Detection System Integrating Text, Speech, and Video: Deep Learning Practice Based on DAIC-WOZ

This project is a deep learning project for depression detection integrating three modalities (text, audio, and video), implemented based on the DAIC-WOZ dataset. Its core goal is to capture the multi-dimensional characteristics of depression through automated methods, providing technical support for early screening and auxiliary diagnosis. The project uses models such as SVM, Random Forest, CNN, and LSTM gating mechanisms to achieve effective fusion and classification of multimodal features. This is an open-source GitHub project developed and maintained by sameer-04062004.

DAIC-WOZ (Distress Analysis Interview Corpus - Wizard of Oz) is a dataset dedicated to mental health research created by the University of Southern California. It includes audio, video, and transcribed text from clinical interviews where participants converse with a virtual interviewer, covering daily life and emotional states. Reasons for choosing this dataset include:

1. Data integrity: Contains three modalities simultaneously, suitable for multimodal research;
2. Clinical annotations: Each sample has professional PHQ-8 depression score labels;
3. Academic recognition: Widely used in mental health AI research, with comparable results;
4. Publicly available: Supports researchers' access applications, promoting collaboration.

The project designs feature extraction methods for different modalities:

• Text modality: Uses SVM and Random Forest to process text features, capturing the language patterns of depressed patients (e.g., more first-person singular pronouns, negative vocabulary, simple sentence structures, etc.);
• Audio modality: Adopts SVM and Random Forest, with pruning optimization to prevent overfitting, extracting speech features (e.g., slower speech rate, less pitch variation, reduced energy, etc.);
• Video modality: Uses CNN to extract spatial features from video frames, capturing facial expressions (e.g., reduced expressions, less eye contact, etc.) and changes in body language.

Single modalities tend to miss information. The core innovation of the project is using LSTM combined with gating mechanisms for sentence-level multimodal fusion:

• Gating mechanism: Dynamically adjusts the weights of each modality, prioritizing reliable ones (e.g., increasing video/text weights when audio is affected by environmental noise);
• Sentence-level fusion: Its advantages include capturing emotional fluctuations in interviews, increasing the number of training samples, and enabling fine-grained localization of abnormal moments.

Potential Application Scenarios

1. Early screening: Preliminary assessment of high-risk groups in communities or online platforms;
2. Auxiliary diagnosis: Providing objective data references for doctors to reduce subjective bias;
3. Efficacy monitoring: Tracking emotional changes during treatment;
4. Telehealth: Serving remote or mobility-impaired populations.

Ethical Considerations

• Not a diagnostic tool: Only for auxiliary screening, cannot replace doctor's diagnosis;
• Privacy protection: Strictly protect sensitive voice/video data;
• Informed consent: Users must clearly understand data usage and participate voluntarily;
• Avoid labeling: Do not use algorithm outputs as fixed labels;
• Fairness: Verify the model's performance across different populations.

Current Limitations

1. Data scale: DAIC-WOZ has a limited sample size, and generalization ability needs verification;
2. Annotation subjectivity: PHQ-8 scores still have certain subjective factors;
3. Real-time performance: Sentence-level processing is difficult to meet real-time application needs;
4. Cross-dataset validation: Need to test the effect on independent datasets.

Future Directions

1. Introduce Transformer architectures (e.g., BERT, Wav2Vec) to improve feature extraction capabilities;
2. Use self-attention mechanisms to complement LSTM and capture long-distance dependencies;
3. Self-supervised learning: Use unlabeled data for pre-training to reduce reliance on labeled data;
4. Interpretability: Develop visualization tools to understand model decisions;
5. Multi-task learning: Predict depression severity, anxiety levels, etc., simultaneously.

Summary

This project demonstrates the application potential of AI in the mental health field. Multimodal fusion is more robust and accurate than single modalities. For learners, it is an excellent introductory project for multimodal learning; for researchers, it provides an extensible technical framework. It is necessary to keep ethical boundaries in mind to ensure that technology serves people.