In-depth Analysis of Music Emotion Recognition: Evolution of Multimodal Technologies from CNN to Transformer

Music Emotion Recognition (MER) is an interdisciplinary field that combines music information retrieval, signal processing, machine learning, and psychology. Based on an open-source review on GitHub, this article systematically sorts out the applications of deep learning in MER, covering emotion modeling, feature extraction, architecture evolution (CNN/BiLSTM/Transformer), multimodal fusion, challenges, and applications, providing readers with a panoramic guide.

Emotion Modeling

• Discrete Model: Divided into mutually exclusive categories such as happiness, sadness, anger, etc. It is intuitive but difficult to describe mixed emotions
• Dimensional Model: Valence-arousal 2D space, capturing emotional gradients (e.g., high arousal + high valence = excitement)

Feature Extraction

• Traditional Handcrafted Features: Mel spectrogram, MFCC, rhythm (BPM), timbre (spectral centroid), harmonic features
• Deep Features: CNN automatically extracts hierarchical audio features from Mel spectrograms

CNN

Treat Mel spectrograms as images to capture local patterns (e.g., chord shapes). Small convolution kernels + deep networks yield better results

BiLSTM

Models temporal dependencies, captures past/future context bidirectionally, suitable for judging phrase-level emotions

Transformer

Self-attention mechanism processes long-distance dependencies in parallel, capturing melodic echoes

Hybrid Architecture

CNN front-end extracts local features + Transformer back-end models global structure, achieving optimal performance

Multimodal Fusion

• Audio Modality: Acoustic cues (melody/harmony/vocals)
• Text Modality: Lyric sentiment polarity (semantic representations generated by BERT)
• Fusion Strategies: Early (feature concatenation), late (decision fusion), attention fusion

Common Datasets

DEAM (Dynamic Emotion Annotation), RAVDESS (controlled experiments), EmoMusic (pop songs), CH-818 (Chinese)

Evaluation Metrics

MSE/R² for dimensional models, accuracy/F1-score for discrete models

• Subjectivity: Emotional perception varies from person to person; need to model diversity
• Cultural Differences: Existing datasets are mainly Western; cross-cultural models need development
• Fine-grained Emotions: Difficult to identify subtle states like nostalgia/awe
• Real-time Processing: Need to improve model efficiency to support recommendation/adaptive scenarios
• Interpretability: The black-box decision process of deep learning needs transparency

• Music Recommendation: Emotion-matched recommendations (e.g., focus/boosting spirit)
• Automatic Generation: Guide AI composition (game/film soundtracks)
• Music Therapy: Assist in selecting therapeutic music and quantifying effects
• Copyright Management: Classify music libraries by emotion
• Affective Computing: Combine with facial/voice data to fully understand user emotions

MER is a bridge between AI and humanities/art, progressing rapidly from handcrafted features to multimodal fusion. The open-source review provides a systematic framework; future multimodal large models/self-supervised learning will promote more delicate emotional perception. Entry suggestions: Reproduce classic papers, explore multimodal/cross-cultural topics, and deeply explore the essence of human emotions.