Emotion Data Studio: A Desktop Tool for Multimodal Emotion Recognition Data Mining

Emotion Data Studio (EDS) is a desktop data mining tool designed specifically for multimodal emotion recognition models, aiming to solve pain points such as time-consuming construction of high-quality emotion recognition datasets and heavy annotation workloads. It integrates full-process features including video import, scene segmentation, face detection, audio analysis, and AI emotion annotation, providing a one-stop data preparation solution for researchers and data scientists.

Multimodal emotion recognition judges emotional states by integrating signals such as facial expressions and voice intonation, which is an important direction in the AI field. However, building high-quality datasets has pain points like time-consuming data collection, heavy annotation work, and inconsistent quality. EDS emerged to provide a complete pipeline from video import to training set export, especially suitable for quickly building datasets for multimodal emotion recognition models.

EDS is designed with features covering the entire data preparation process:

1. Video Import: Supports YouTube download or local file import, making it easy to obtain emotional video materials;
2. Scene Segmentation: Uses PySceneDetect to automatically split long videos into independent emotional clips;
3. Face Detection and Tracking: Adopts SCRFD and ByteTrack technologies to focus on the target person's expressions;
4. Audio Analysis: Extracts MFCC features and converts speech to text via Whisper, laying the foundation for multimodal fusion.

EDS's AI annotation function uses a multi-model integration voting mechanism, combining four professional models:

• HSEmotion (facial expression recognition), DeepFace (multi-dimensional facial analysis), PhoBERT (text emotion), Wav2Vec2 (speech emotion);
• Advantages: Integrates the expertise of different models to improve annotation accuracy; marks uncertain cases for manual review when there are conflicts;
• Quality scoring module: Automatically scores and filters high-quality samples, considering factors such as face clarity and audio quality.

AI annotation requires manual review to ensure quality. The EDS review studio provides:

• Intuitive interface + keyboard shortcuts for efficient browsing of annotations;
• Batch operations and label management for systematic annotation;
• Real-time saving of annotation results to the local SQLite database to ensure data security.

• Interface: Built with PySide6 for a native desktop interface, based on Qt6 to provide a smooth experience;
• Backend: Developed in Python, with the AI pipeline based on PyTorch, integrating models such as Whisper and DeepFace;
• Dependencies: Video processing using FFmpeg/PySceneDetect, YouTube download using yt-dlp;
• Database: Local SQLite, with cloud synchronization supporting PostgreSQL;
• Deployment: Uses PyInstaller + Inno Setup to generate Windows installers, making it easy for non-technical users to use.

EDS is applicable to:

• Academic Research: Quickly build domain-specific datasets (e.g., emotional expressions of specific cultures/age groups);
• Commercial Applications: Analyze emotional trends in customer feedback videos to optimize products and services;
• AI Developers: Lower the threshold for project initiation, no need to build data pipelines from scratch, and iterate models quickly.

Emotion Data Studio is a fully functional and well-designed data preparation tool for multimodal emotion recognition, integrating video processing, AI annotation, manual review, and other features to provide a one-stop solution. For developers and researchers engaged in related studies, it is a worth-trying open-source tool.