Multimodal Emotion Recognition System: Intelligent Emotion Analysis Integrating Speech and Text

This article introduces a multimodal emotion recognition system based on MFCC speech features and BERT text embeddings, and discusses the application and effects of fusion learning in emotion analysis. The system uses a dual-branch architecture to process speech and text inputs, integrating complementary information to improve emotion recognition accuracy, and is suitable for scenarios such as human-computer interaction and mental health monitoring. The project is from GitHub user umasri15, who released Multimodal-Emotion-Recognition on May 24, 2026.

Emotion recognition is widely used in the AI field, but traditional single-modal methods have limitations: pure text cannot capture emotions that rely on intonation such as sarcasm, and pure speech struggles to understand the impact of semantic context. Human emotional expression is multimodal, requiring the integration of acoustic features (intonation, speech rate, etc.) and linguistic features (vocabulary, sentence structure, etc.). Multimodal fusion can achieve more accurate and robust recognition.

The system adopts a dual-branch architecture:

• Speech Branch: Uses MFCC (Mel-Frequency Cepstral Coefficients) to extract features, simulating human auditory characteristics, capturing emotional information such as timbre and pitch changes, with noise robustness and dimensional efficiency.
• Text Branch: Uses a pre-trained BERT model to extract semantic embeddings, understanding subtle differences in contextual semantics (such as polarity reversal in negative sentences).
• Fusion Strategy: Instead of simple concatenation, it learns weight allocation for different modal features through a fusion layer, achieving optimal integration at the decision level, which is more flexible than early/late fusion.

The project uses the TESS (Toronto Emotional Speech Set) dataset for training and evaluation, which includes seven emotion categories: anger, fear, happiness, surprise, sadness, disgust, and neutral. The multi-classification setup is closer to real-world scenarios, and the dataset's balance prevents the model from being biased toward dominant categories.

The system can be applied to:

1. Intelligent Customer Service: Real-time analysis of customer speech and text to identify emotions and remind customer service to adjust strategies;
2. Mental Health Monitoring: Long-term analysis of users' speech and text communication to capture emotional change trends;
3. Educational Assistance: Analyze students' speech responses and assignments to identify emotions such as confusion/frustration;
4. Human-Computer Interaction Optimization: Voice assistants/chatbots adjust response strategies to enhance interaction experience.

The current system has room for improvement:

1. Dataset Scale: The TESS dataset is small; larger-scale and diverse data are needed to improve generalization ability;
2. Modal Alignment: Optimize the time alignment mechanism between speech and text to capture temporal correlations;
3. Cross-Language Support: Extending to other languages requires corresponding pre-trained models and datasets;
4. Real-Time Performance: Optimize inference speed (especially for BERT) to adapt to real-time scenarios.

This project demonstrates an effective method to improve emotion recognition performance by integrating speech and text information, achieving accurate recognition of seven basic emotions through MFCC, BERT, and a fusion layer. The code is complete and modular, providing a reference for research and applications in the field of affective computing. With the development of multimodal technology, such fusion methods will play a role in more scenarios.