MEDS: A Multimodal Emotion Detection System Bridging the 'Emotional Gap' in Voice Interaction

MEDS is an innovative multimodal emotion detection system. By integrating speech-to-text (Whisper) and audio feature extraction (Librosa) technologies, combined with the Oumi small language model, it identifies discrepancies between users' utterances and their true emotions, solving the 'emotional gap' problem where AI voice assistants fail to perceive real emotions. It features privacy-first design and low latency, bringing emotional understanding capabilities to voice interactions.

Traditional voice AI relies only on text input, missing acoustic features like intonation and speech rate (in human communication, language content accounts for only 7%, while sound features account for 93%), leading to failure in perceiving users' true emotions. This limitation is particularly prominent in scenarios like mental health support and customer service—for example, when a depressed user says 'I'm fine', AI cannot detect the hidden pain.

MEDS adopts an 'emotion + semantic fusion' approach: 1. The speech-to-text layer uses the Whisper model for accurate recognition; 2. The audio intelligence layer extracts features like pitch, energy, timbre, and speech rate via Librosa; 3. The intelligent reasoning layer uses a fine-tuned Oumi small language model (local processing, low latency, resource-efficient) to comprehensively analyze text and audio, identifying complex emotions such as 'false positivity'. The system uses a front-end and back-end separation architecture: the front-end is a real-time visualization dashboard, and the back-end is coordinated via Flask.

MEDS is applicable in multiple scenarios: mental health support (identifying emotional crises to trigger care), customer service (monitoring customer emotion escalation in conversations), educational counseling (analyzing student status to adjust teaching), and smart homes (recommending content based on emotions).

MEDS was developed by the five-member Team pENTEX: Mannat Sharma was responsible for architecture and documentation, Chaitali Mahajan for front-end, Gurshant Singh Mohal for AI pipeline integration, Soham Sahu for infrastructure, and Vrinda Kaushal for DevOps and Git management.

Current challenges: Data privacy compliance, cross-cultural differences in emotion recognition, and real-time performance optimization. Future plans: Expand support for multilingual dialects, integrate facial expression analysis, develop lightweight models for mobile devices, and build emotion datasets to promote research.

MEDS represents the evolution of voice AI from understanding 'what was said' to perceiving 'how it was said' and 'how the speaker feels', providing a feasible path to bridge the emotional gap in human-computer interaction. Future AI assistants will have both IQ and EQ, understanding the emotional world behind words.