SALSTM-LWARO: A New Multimodal Sentiment Recognition Framework Breaking Through Local Optima

This article introduces the SALSTM-LWARO framework, which combines self-attention LSTM with the Lightweight Weighted Adaptive Optimization algorithm (LWARO). It effectively solves the problem of traditional models falling into local optima during hyperparameter optimization, achieving a sentiment recognition accuracy of 97.73%, and is suitable for processing text, audio, and video multimodal data.

In today's era of increasingly frequent human-computer interaction, sentiment recognition technology plays a key role in fields such as intelligent customer service, online education, and auxiliary medical care. However, traditional deep learning model training tends to fall into local optima, especially in multimodal tasks where feature fusion of text, audio, and video is intertwined with hyperparameter tuning, leading to an exponential expansion of the search space and difficulty in finding the global optimal solution.

The framework adopts a three-layer progressive architecture: the feature extraction layer processes three modal data (BERT is used for text to capture semantics, MFCC for audio to convert frequency spectrum, and ResNet for video to extract facial expression dynamics); the middle layer introduces self-attention enhanced LSTM (SA-LSTM), which dynamically adjusts the feature weights of time steps to solve the problem of long-sequence information attenuation.

The LWARO algorithm introduces an adaptive weight adjustment mechanism: during iteration, it dynamically adjusts the search step size and direction weights based on the quality of the solution—when falling into local optima, it increases the exploration weight; when approaching the global optimum, it enhances local search. Compared with traditional genetic algorithms and particle swarm optimization, it has low computational overhead and does not require maintaining a large population, making it suitable for edge device deployment.

In tests on the SAVEE dataset (480 audio-visual clips, six emotions), the framework achieved an accuracy of 97.73%, outperforming traditional methods such as SER-XGBoost; ablation experiments showed that removing LWARO reduced the accuracy by about 4 percentage points; it performed stably in cross-speaker scenarios, proving that the sentiment features are speaker-independent.

The framework has broad application prospects: real-time monitoring of driver fatigue and emotions in smart cockpits; auxiliary analysis of patients' non-verbal emotional cues in telemedicine; evaluation of online learners' engagement and confusion in educational technology; open-source release lowers the threshold for multimodal sentiment recognition technology, allowing developers to quickly adapt to domain-specific data.

With the development of lightweight models and edge computing, efficient frameworks like SALSTM-LWARO are expected to be implemented in more real-time scenarios. Sentiment recognition technology is moving from the laboratory to daily life, becoming an important cornerstone of natural human-computer interaction.