Deep Learning-Based Speech Emotion Recognition System: Complete Implementation from Audio Signals to Emotion Classification

This article introduces an end-to-end speech emotion recognition project built with PyTorch. Using MFCC feature extraction and a multi-layer perceptron neural network, it achieves automatic recognition of eight emotions (neutral, calm, happy, sad, angry, fearful, surprised, disgusted) in speech with a validation accuracy of 69.10%. The project originates from the CodeAlpha Machine Learning Internship, covering all stages including data preprocessing, feature extraction, model training, and inference deployment. The code is maintained by Ahmed Gul and published on GitHub (link: https://github.com/Ahmed-Gul16/CodeAlpha_Emotion-Recognition-from-Speech-).

Speech Emotion Recognition (SER) is an important direction in the field of human-computer interaction, enabling machines to understand human emotions. Traditional speech recognition only focuses on text conversion and ignores emotional cues such as intonation and speech rate. SER technology can be applied in scenarios like intelligent assistants, customer service robots, and mental health monitoring to achieve more natural interactions. This project aims to build a deep learning system that automatically recognizes emotions from speech audio, with a modular design covering all complete stages.

The project uses the RAVDESS dataset for training and validation. Key features of this dataset:

• Recorded by 24 professional actors (12 male, 12 female)
• 8 emotion categories: neutral, calm, happy, sad, angry, fearful, surprised, disgusted
• Professional studio environment with a sampling rate of 48kHz (later downsampled to 16kHz)
• Manually verified accurate emotion labels Its professionalism and standardization provide a reliable foundation for model training.

Speech signals need to be converted into features before being input into the neural network. The project uses MFCC (Mel-Frequency Cepstral Coefficients), with steps including pre-emphasis, framing and windowing, FFT, Mel filter bank, logarithmic operation, and DCT. The librosa library is used to extract 40-dimensional MFCC features, capturing spectral envelopes and emotion-related information (pitch, timbre, etc.).

The model implements an MLP based on PyTorch:

• Input layer: 40-dimensional MFCC features
• Hidden layers: Fully connected layers + ReLU activation + Dropout regularization
• Output layer: 8 neurons (corresponding to 8 emotions) + Softmax probability distribution Training strategy: Cross-entropy loss, Adam optimizer, learning rate scheduling, and early stopping mechanism. After 50 epochs of training, the validation accuracy reaches 69.10%. Considering the difficulty of 8-class classification, the result is solid.

Inference supports custom .wav files:

1. Audio loading (using the soundfile library)
2. Preprocessing (resampling, normalization, framing)
3. MFCC feature extraction
4. Model inference to get probability distribution
5. Output predicted emotion and confidence It can be integrated into scenarios like real-time emotion monitoring, customer service quality assessment, and mental health screening.

Experimental results:

• Overall accuracy: 69.10% (8 classes)
• High-intensity emotions (anger, fear) are recognized well, while similar emotions (neutral, calm) are easily confused
• Converges after 30-40 epochs Future expansion:

1. Model upgrade (CNN/LSTM)
2. Data augmentation (SpecAugment)
3. Multimodal fusion (facial expressions, text)
4. Lightweight models for edge deployment This project provides an excellent learning case for speech processing beginners. Future multimodal/self-supervised learning is expected to improve accuracy.