EmotionNet: A Multimodal Sentiment Analysis Project Exploring Text and Speech Emotion Recognition

EmotionNet is a multimodal emotion recognition neural network system that combines text and speech data. This article introduces its background, technical architecture, comparative experiments with large language models, application scenarios, limitations, and future directions, exploring the value of multimodal fusion in emotion recognition.

Emotion recognition technology is widely used in fields such as human-computer interaction, customer service, and mental health monitoring. Traditional analysis is limited to a single modality, while human emotional expression includes words as well as sound features like intonation and speech rate. EmotionNet originated from a course project at the Catholic University of Lisbon, aiming to integrate text and speech to build a more accurate and robust emotion recognition system.

The project is built using Python and TensorFlow, with a multimodal neural network at its core. It processes heterogeneous data: text is converted into word embedding sequences, while speech features such as Mel spectrograms or MFCCs are extracted. After processing via CNN/RNN, the features from the two modalities are fused at an early, middle, or late stage to address challenges in alignment, fusion, and joint training.

The project compares traditional neural networks with LLMs: 1. Specialized architectures may perform better in specific tasks and resource-constrained scenarios; 2. Traditional models require less training data to converge, while LLMs need more samples; 3. Specialized models allow easy feature analysis, whereas the black-box nature of LLMs makes it difficult to understand their decision-making process.

Multimodal emotion recognition is applied in: real-time adjustment of communication strategies in customer service; assessment of learner engagement in education; auxiliary mental health screening in healthcare. For developers, the project provides a complete technical reference implementation (including data preprocessing, model definition, etc.) and serves as a learning resource.

As a course project, it has limitations such as dataset size and model complexity; production-level applications need to consider real-time performance and privacy. Future directions include: introducing Transformers to replace CNN/RNN, using self-supervised pre-training to reduce reliance on annotations, and expanding to video modality to integrate facial expressions.

EmotionNet represents the multimodal evolution trend in emotion recognition, combining text and speech to capture rich emotional cues; the comparative experiments provide a basis for technology selection; it is a reference project for developers new to multimodal deep learning.