Multimodal Emotion Recognition System: Intelligent Sentiment Analysis with Speech and Text Fusion

Original Author/Maintainer: Abel-Jacob Source Platform: GitHub Project Link: https://github.com/Abel-Jacob/multimodal-emotion-recognition Release Date: May 28, 2026

This project builds a multimodal emotion recognition system based on the TESS dataset, fusing speech (CNN+BiLSTM+Attention) and text (DistilBERT) features to address the limitations of single-modal systems, improve emotion classification accuracy, and has broad application prospects in human-computer interaction.

Emotion recognition is a core technology in human-computer interaction. Traditional single-modal systems (either speech or text) cannot fully capture human multimodal emotional expressions. Multimodal systems reduce misjudgment rates and more accurately restore real emotional states by analyzing speech and text simultaneously.

Speech Processing Pipeline: CNN + BiLSTM + Attention

CNN extracts local time-frequency features, BiLSTM models temporal dependencies, and the attention mechanism enables "selective listening"—the speech pipeline achieves a test accuracy of 91.81%.

Text Processing Pipeline: DistilBERT Embedding

DistilBERT (a lightweight variant of BERT) retains 95% of the performance while increasing inference speed by 60% and reducing volume by 40%, capturing text semantics and emotional cues.

Fusion Strategy: Multimodal Feature Joint Modeling

Deep fusion allows speech and text features to interact and enhance each other—text supplements when there is noise, and speech corrects ambiguities, making it more robust than single-modal systems.

The TESS dataset (recorded by elderly women from the University of Toronto) is used, containing 7 emotion categories (anger/fear/happiness/sadness/surprise/disgust/neutral), with 200 samples per category. The dataset is divided into training/validation/test sets, and data augmentation (adding noise, adjusting speech rate) is used during training to improve generalization ability.

Multimodal emotion recognition application scenarios:

• Intelligent Customer Service: Monitor user frustration and automatically transfer to human agents;
• Online Education: Analyze student emotions to adjust teaching strategies;
• Mental Health: Assist in early screening of emotional disorder symptoms;
• In-Vehicle Systems: Monitor driver emotions to prevent accidents;
• Interactive Robots: "Read emotions" to provide thoughtful services.

The project verifies the effectiveness of multimodal fusion—speech and text fusion achieves a 1+1>2 effect, and can be extended to modalities such as facial expressions and physiological signals. In the future, with the development of large models, accuracy and generalization ability will improve; attention should be paid to user privacy protection issues.