Watching Movies Like Humans: Egocentric Perspective Emotion Understanding for Embodied Companion Robots

This article focuses on the problem of emotion understanding of movies from an egocentric perspective for embodied companion robots. The core finding is that existing models trained on movie shots experience a sharp performance drop in real-world viewing scenarios, while the EgoScreen-Emotion (ESE) benchmark dataset proposed by the research team can significantly improve model robustness. The study emphasizes the importance of domain-specific data and long-context multimodal reasoning for achieving human-robot emotional empathy.

Embodied robots cannot directly access movie source files and can only watch the screen through cameras, leading to multiple domain shifts between the egocentric screen perspective and movie shots:

1. Perspective distortion: Camera angle/height causes screen tilt and deformation
2. Scale variation: Distance affects the proportion of the screen in the field of view
3. Lighting changes: Reflections, glare, or ambient light pollution
4. Environmental interference: The field of view includes irrelevant information such as rooms and furniture These differences cause a significant drop in the performance of existing models in real-world scenarios.

Data Collection

• Content selection: 224 movie trailers with high emotional density and diverse genres
• Collection setup: Head-mounted/fixed cameras simulate robot perspectives, collected under different distances, angles, and lighting conditions, with real environments recorded
• Result: 28,667 time-aligned keyframes

Annotation Strategy

A confidence-aware multi-label protocol is adopted:

• Multi-label: Allows multiple emotions to be annotated for one sample
• Multi-annotator: Captures subjectivity
• Confidence score: Reflects the certainty of judgment A rich emotional annotation set is generated.

Four-Modal Fusion Architecture

1. Temporal visual evidence: Processes continuous frame sequences to capture emotional changes, visual rhythm, etc.
2. Narrative summary: Introduces text information such as plot synopses and genre tags to assist in understanding narrative positions
3. Compressed historical context: Maintains emotional memory vectors and retrieves relevant historical segments
4. Audio cues: Extracts acoustic features such as background music and dialogue intonation

Long-Context Modeling

• Local encoding: Splits short segments to extract features
• Global aggregation: Transformer handles segment-level long dependencies
• Adaptive sampling: Uses higher resolution for emotionally rich regions Effectively handles long video sequences.

Key Findings

1. Significant domain gap: Models trained on movie shots see their Macro-F1 drop from 27.99 to 16.69 in egocentric perspective tests, a decrease of over 40%
2. ESE improves robustness: Models trained on ESE are more tolerant to disturbances such as perspective distortion and lighting changes
3. Multimodal fusion is effective: Four-modal fusion (visual, audio, text, historical context) achieves the best performance
4. Competition with closed-source models: The research method can compete with closed-source models like GPT-4V and Gemini on the ESE benchmark Confirms the value of domain-specific data and architectural design.

Core Applications

1. Companion robots: Accompany users to watch movies, perceive emotions, and interact
2. Educational assistance: Detect students' confusion/interest and adjust teaching strategies
3. Health monitoring: Monitor emotional changes of elderly people living alone and issue abnormal alerts
4. Entertainment recommendation: Analyze emotional preferences and recommend suitable content

Deep Significance

The study reveals the impact of differences between AI's perception method and humans' on task performance, which is an important step toward true human-robot empathy. The goal is to enable robots not only to understand movies but also to comprehend the emotional needs of viewers.

Current Limitations

• Data scale: 224 trailers are limited
• Cultural diversity: Mainly Western movies
• Real-time performance: Need to optimize real-time processing capabilities
• Multi-user scenarios: Does not cover multi-person social viewing

Future Directions

• Expand data scale and cultural diversity
• Cross-modal pre-training to improve generalization ability
• Personalized adaptation to specific users' emotional patterns
• Explore emotional causal reasoning
• Support interactive emotional communication Provides directions for the development of emotion understanding in embodied AI.