EmotionLayer: A Multimodal Empathetic Voice Assistant Architecture Integrating Speech Emotion Recognition and Large Language Models

EmotionLayer is an innovative multimodal architecture developed by the research team at the University of Milan. By integrating Speech Emotion Recognition (SER) with Large Language Models (LLM), it addresses the "emotional blind spot" issue of traditional voice assistants, enabling dual understanding of both the content and emotion of users' utterances and generating empathetic responses. The architecture adopts a layered design and features modularity, open-source availability, etc., providing a new solution for emotionally intelligent human-computer interaction.

Most current voice assistants only understand the content of commands and ignore emotional nuances, leading to mechanical responses that limit the naturalness and experience of interaction. Targeting this pain point, EmotionLayer aims to build a voice assistant with emotional perception capabilities, deeply integrating the semantic understanding of SER and LLM to capture both what the user "said" and "how they said it".

EmotionLayer adopts a layered architecture: the bottom layer extracts acoustic features such as pitch, speech rate, and energy; the middle layer maps emotion categories (e.g., happiness, sadness, etc.) through a Transformer-based SER engine; after speech-to-text conversion, the text is sent to the LLM layer along with emotion labels, which adapts to emotional scenarios via dynamic prompt templates and performs emotion consistency checks. For SER implementation, a multi-scale feature fusion strategy is used, integrating datasets like IEMOCAP, and improving generalization ability through data augmentation and multi-label filtering.

EmotionLayer can be applied in fields such as mental health (emotional support robots), customer service (identifying customer emotions for priority intervention), education (adaptive intelligent tutoring systems), etc., enhancing interaction naturalness and user experience, and creating value for enterprises and users.

1. Deep multimodal fusion: Acoustic and semantic information are intertwined to achieve cross-modal joint推理; 2. Modular design: Functional encapsulation allows flexible combination; 3. Open-source and open: A permissive license encourages community contributions and secondary development.

Current limitations: Limited granularity of emotion recognition (only basic categories), insufficient cross-language and cultural adaptability, and real-time performance needing optimization. Future plans: Introduce multilingual data, explore cultural perception modeling, optimize model lightweighting; evolution directions include multimodal emotion recognition, personalized emotional memory, and emotional feedback loops.

EmotionLayer is an important step in the evolution of voice assistants toward emotional intelligence, proving that machines can "understand" emotions. With technological progress, such open-source projects will promote more natural and warm human-computer interaction, providing exploration directions for researchers and developers.