Application of Cross-Modal Attention Mechanism in Depression Detection: Analysis of a Lightweight Multimodal Deep Learning Framework

This article analyzes a depression detection study based on cross-modal attention fusion mechanism. The study integrates three modal information (audio, visual, text), uses data from 97 subjects in the DAIC-WOZ dataset, achieves an 80% detection accuracy, proposes a lightweight multimodal deep learning framework, and wins the Best Demo Award at ICITACEE 2025. The core innovation lies in capturing complex interactions between modalities through multi-head cross-modal attention, providing an effective solution for automated depression detection.

Depression is a common global mental health issue. Traditional diagnosis relies on subjective assessment and self-reporting, which has problems such as delay and strong subjectivity. AI technology has promoted multimodal automated detection to become a hot topic. The team from Amikom Yogyakarta University in Indonesia published the study at ICITACEE 2025, proposing a lightweight framework that integrates three modal information to achieve efficient detection and won the Best Demo Award.

The study uses the DAIC-WOZ dataset, developed by the USC ICT SimSensei project, which contains clinical interview videos of 189 participants with PHQ-8 depression labels. The dataset has multimodal characteristics (audio, facial video, text transcription), supporting the exploration of complementary information. Due to storage limitations, the study only used data from 97 participants, which limits the generalization ability to some extent.

Audio Modality: Extract MFCC (spectral envelope), COVAREP (vocal fold vibration), and formant features; the subnetwork uses SimpleRNN + Dropout (0.3) + L2 regularization. Visual Modality: OpenFace extracts Action Units (FACS), eye gaze, and head pose; the subnetwork uses Conv1D + max pooling + fully connected layer. Text Modality: BERT-base generates semantic embeddings; the subnetwork adds fully connected layer, BatchNorm, and Dropout.

The core innovation is multi-head cross-modal attention fusion. Traditional fusion (early/late) is difficult to capture modal interactions. This study uses each modality as a query, others as keys/values, and calculates pairwise modal attention (e.g., audio focuses on visual/text, etc.). It is configured with 2 attention heads (key dimension 16). After fusion, global average pooling is applied, and the result is input to the classification head for binary classification (depressed/non-depressed).

Training Strategy: Nadam optimizer (initial learning rate 1e-5), ReduceLROnPlateau (halve learning rate when loss plateaus); regularization uses Dropout (0.3), L2, and EarlyStopping (stop if no improvement in 10 epochs); handle class imbalance with manual oversampling + SMOTE. Results: 80% accuracy, macro-average F1 = 0.78, weighted F1 =0.81. Class performance: non-depressed (recall 83%, precision 62%), depressed (recall79%, precision92%).

Limitations: Only 97 samples used (limited generalization), binary classification (no distinction of depression severity), no uncertainty quantification. Future Directions: Expand to multi-class classification (distinguish PHQ-8 levels), integrate uncertainty quantification (e.g., Bayesian NN), external dataset validation, enhance interpretability (attention visualization).

Clinical Value: The lightweight model is suitable for remote screening (real-time analysis on mobile/web), clinical auxiliary diagnosis (providing second opinions), and longitudinal monitoring (tracking symptom changes to evaluate treatment efficacy). Open-Source: The code has been open-sourced on GitHub (MIT license), including training notebooks and configurations; the DAIC-WOZ dataset needs to comply with its own license terms.