NeuroSense: A Production-Grade Multi-Modal Emotion Recognition System—Practical Microservice Architecture Integrating Text, Speech, and Deep Learning

NeuroSense is a production-grade multi-modal emotion recognition system that integrates text and audio inputs using RoBERTa and Wav2Vec2 models respectively. It achieves over 90% recognition accuracy via weighted late fusion and is built on a micro-service architecture with FastAPI and Streamlit, providing a reference engineering paradigm for the emotion computing field.

Emotion computing is shifting from academic research to practical applications. However, single-modal emotion recognition is limited by insufficient information—text may hide true emotions, voice intonation conveys unspoken feelings, and facial expressions can be disconnected from context. NeuroSense's core insight is that true emotion understanding requires combining text semantics and acoustic features to mimic human emotional perception.

Building a production-ready multi-modal emotion system faces several challenges:

1. Modal heterogeneity: Text (discrete symbols) and audio (continuous waveforms) have different feature spaces and information densities.
2. Time alignment: Speech-text transcription may be misaligned, requiring precise timing for effective fusion.
3. Model complexity vs. inference efficiency: Balancing accuracy and low latency for production.
4. Data scarcity: High-quality multi-modal labeled data is rare, limiting end-to-end training.

NeuroSense uses a micro-service architecture with three core layers:

• Streamlit Frontend: Interactive web interface for uploading audio/text and viewing results, communicating with backend via REST API.
• FastAPI Backend: Asynchronous RESTful API with PyTorch-based inference (GPU-accelerated) and four endpoints (health check, text/audio/multi-modal analysis).
• Supabase PostgreSQL: Stores predictions (input mode, dominant emotion, confidence, probability distribution, inference latency) for analysis and monitoring.

NeuroSense employs a dual-branch model:

• Text Branch: Uses j-hartemann/emotion-english-distilroberta-base (RoBERTa-based) to map text to 7 Ekman emotions, achieving ~86% accuracy on MELD.
• Audio Branch: Uses superb/wav2vec2-base-superb-er (Wav2Vec2-based) for speech emotion recognition, with ~67% weighted accuracy on IEMOCAP. Both branches run independently, supporting single/multi-modal inputs and future modal extensions.

NeuroSense uses weighted late fusion at the decision layer: text (55%) and audio (45%) probability distributions are weighted and summed for final prediction. This strategy offers modularity (update branches independently), handles missing modes (degrade to single-modal), and is easy to maintain.

Key engineering practices:

• Model Service: Pre-trained models loaded once at startup to reduce latency.
• Async Processing: FastAPI's async design improves throughput.
• Error Handling: Degrades to single-modal if one mode fails.
• Logging: All predictions stored in Supabase for performance analysis. Deployment: Render.com with render.yaml (IaaC), environment variables for sensitive data, scalable micro-services.

Applications: Customer service (real-time emotion monitoring), mental health (clinical auxiliary tool), education (student feedback analysis), market research (focus group emotion extraction). Limitations: English-only, no context modeling, cultural bias, privacy concerns. Future: Multi-language support, context-aware analysis, cross-cultural adaptation, stronger privacy measures.