Multimodal Fusion and LLM Empowerment: A New Intelligent Medical Solution for Depression Detection

This project innovatively combines facial expression features with the text processing capabilities of large language models (LLMs) to build a bimodal depression detection system. By fusing visual and language information, it achieves more accurate depression severity assessment than unimodal methods on the E-DAIC dataset, providing a new direction for intelligent medical auxiliary diagnosis.

Depression affects over 280 million people worldwide. Traditional diagnosis relies on doctors' subjective assessments and patients' self-reports, which have issues like delays and strong subjectivity. The development of AI technology has promoted multimodal automated detection as a hot topic—integrating information such as facial expressions, voice, and text can capture symptoms more comprehensively, and the emergence of LLMs provides new possibilities for deep text understanding.

Visual Analysis:

Use OpenFace to extract features such as facial action units (AUs), head pose, eye tracking, and facial key points, then model the dynamic temporal patterns of expressions via LSTM.

Text Processing:

Use GPT-3.5 Turbo to generate interview text completion, then perform depression classification via DepRoBERTa (a mental health pre-trained RoBERTa variant), outputting three types of results.

Fusion Strategy:

Feature-level fusion of visual and text features, using an SVR regression model to predict PHQ-8 scores, with end-to-end training to optimize the overall system.

Dataset:

Based on the Extended DAIC (E-DAIC) dataset, which includes clinical interview videos and PHQ-8 scores, divided into training/validation/test sets to ensure reliability.

Evaluation Metrics:

Classification accuracy, MSE/MAE for PHQ-8 prediction, macro-average/weighted average F1 scores.

Implementation:

Modular architecture (data/script/source code directories), three-stage training (video model → text model → multimodal fusion), requiring an OpenAI API key for text processing.

1. Remote Screening: Analyze video interviews to achieve contactless preliminary assessment, suitable for patients in remote areas or with limited mobility;
2. Clinical Assistance: Provide objective data to assist doctors in diagnosis, reducing missed diagnoses and misdiagnoses;
3. Treatment Monitoring: Track changes in expressions and language to evaluate treatment effects.

Advantages:

• LLM-empowered text understanding to capture deep semantics and emotions;
• Visual + text complementarity, combining non-verbal behavior with subjective descriptions;
• Fusion strategy enhances interpretability.

Challenges:

• Data privacy protection;
• Generalization ability under cultural differences needs verification;
• Effectiveness in real clinical environments requires large-scale validation.

Future Directions:

Integrate voice modality, optimize real-time detection, develop personalized models;

Open-Source Value:

Modular design facilitates reproduction and expansion, provides references for multimodal mental health AI research, and supports community contributions of new methods.