Neurosense: A Multimodal AI Real-Time Stress Detection System

Neurosense is a multimodal AI real-time stress detection system that integrates facial expressions, voice features, text content, and questionnaire data. It aims to address the limitations of traditional stress assessment, which are subjective and non-real-time. By fusing multiple modalities, it enhances detection accuracy and robustness, and is applied in various scenarios such as the workplace, clinical settings, and education. It emphasizes privacy protection and user experience, providing an innovative solution for mental health monitoring.

Limitations of Single Modality

Traditional stress detection relies on physiological indicators (easily disturbed), questionnaires (subjective), or expression analysis (easily faked), making it difficult to fully reflect the real state.

Advantages of Multimodal Fusion

Multi-channel information complements and verifies each other, capturing signals that are hard to fake (e.g., voice changes, text emotions), thus improving detection reliability.

Need for Real-Time Monitoring

Stress changes dynamically; real-time tracking can identify peaks and troughs, providing a basis for timely intervention for high-stress groups, chronic disease patients, etc.

Core Modules

• Facial Emotion Recognition: Analyzes micro-expressions and action units (e.g., frowning), non-invasive with local processing to protect privacy;
• Voice Emotion Analysis: Extracts acoustic features such as speech rate and pitch, supporting unobtrusive detection of short segments;
• Text Emotion Analysis: Mines emotional tendencies and stress-related vocabulary from chat/social text to trace historical trends;
• Questionnaire Assessment: Integrates scales like PSS/DASS to calibrate the model's baseline.

Fusion Algorithm

Confidence-weighted fusion: Dynamically adjusts modality weights based on the environment (e.g., reducing facial weight in low light), handles modality conflicts, and marks uncertainties.

• Streamlit Dashboard: Intuitively displays stress levels, historical trends, and modality contributions, supporting real-time updates;
• Instant Alerts and Recommendations: Abnormal stress triggers customized alerts and recommends evidence-based stress reduction strategies;
• PDF Report Generation: Exports trend analysis, risk identification, and improvement suggestions to assist health management;
• Privacy Protection: Local data processing, encrypted storage, and user control over data permissions;
• Fast Response: Completes data collection to result output within 2 seconds, ensuring real-time performance.

• Workplace Management: Integrates with corporate health systems to prevent employee burnout, reduce turnover and medical costs;
• Clinical Assistance: Provides objective data for doctors, supplements consultation assessments, and guides treatment plan adjustments;
• Educational Scenarios: Establishes mental health early warning mechanisms for students to help with study stress management;
• Personal Health: Tracks the correlation between stress and lifestyle to develop personalized management strategies.

• Data Annotation: Uses weak supervision (questionnaires), crowdsourcing + expert review, and semi-supervised learning to address subjectivity issues;
• Cross-Individual Differences: Collects baseline data during initial use and continuously learns user patterns for personalized calibration;
• Environmental Adaptability: Adaptive algorithms handle light/noise interference, and robust design is compatible with different devices;
• Ethical Bias: Diverse training data and fairness testing ensure reliability for all groups.

• Wearable Integration: Integrates physiological signals like HRV and GSR to improve detection accuracy;
• Predictive Analysis: Predicts stress peaks through time-series patterns for early warning;
• Personalized Intervention: Recommends suitable stress reduction methods based on user profiles;
• Group Monitoring: Aggregates group data under privacy protection to assist organizational decision-making.

Neurosense demonstrates the potential of AI in the field of mental health. With multimodal fusion, real-time monitoring, and privacy protection as its core, it assists professional medical care and helps the public manage stress. The project adheres to a humanistic technology perspective, promoting technology to serve user well-being and help improve quality of life.