Social-MMU: A New Benchmark for Evaluating Social Intelligence of Multimodal Large Language Models

Social-MMU is a comprehensive benchmark framework dedicated to evaluating the social intelligence capabilities of multimodal large language models, aiming to fill the gap in social-level capability assessment in traditional multimodal evaluations. By designing multi-dimensional test tasks covering emotion recognition, social situational reasoning, intent inference, empathetic response, etc., this benchmark promotes the standardization of performance evaluation for AI in social scenarios and helps build AI systems with higher social sensitivity.

With the rapid development of multimodal large language models such as GPT-4V, Gemini, and Claude, AI systems have made significant progress in basic capabilities like visual understanding and text generation. However, their performance in complex social scenarios (e.g., understanding human social intentions, emotional states, and situational contexts) still needs to be evaluated. Traditional multimodal benchmarks mainly focus on basic visual tasks such as object recognition and scene description, with relatively weak assessment of social-level capabilities. Thus, the Social-MMU project was born.

Initiated by researcher GordonChen19, the core objectives of Social-MMU include: establishing a standardized and quantifiable evaluation system for social intelligence capabilities; covering a full spectrum of tasks from basic emotion recognition to complex social reasoning; identifying the shortcomings and improvement directions of current multimodal models in social intelligence; and providing evaluation basis for building responsible socially sensitive AI systems.

Social-MMU conducts evaluations around five core dimensions:

1. Emotion Recognition and Understanding: Identify basic and complex emotions in images/videos/texts, and evaluate the ability to recognize emotion intensity and mixed emotional states;
2. Social Situational Reasoning: Understand behavioral norms, role relationships, power dynamics, and cross-cultural social etiquette in specific occasions;
3. Intent Inference and Theory of Mind: Infer others' psychological motivations and social intentions through clues such as behaviors and expressions;
4. Empathy and Appropriate Response: Generate socially appropriate empathetic responses after perceiving others' emotions;
5. Multimodal Information Integration: Effectively integrate multimodal information such as visual and text to form a complete understanding of social situations.

The evaluation design of Social-MMU follows four principles:

• Ecological Validity First: Tasks are close to real social scenarios, with data sourced from real interactions such as daily conversations, social media, and film/television works;
• Multi-level Difficulty: Tasks are distributed with gradient difficulty to distinguish models of different capabilities and reveal bottlenecks;
• Cross-cultural Universality: Cover diverse social scenarios to avoid over-representation of specific cultures;
• Interpretability Evaluation: Focus on the rationality and interpretability of the model's reasoning process, rather than just the correctness of the final answer.

The significance of Social-MMU for multimodal AI research includes:

1. Providing a unified evaluation platform to enable comparability of social intelligence capabilities across different models;
2. Revealing the limitations of current models and guiding future improvement directions;
3. Offering selection references for application developers, suitable for scenarios requiring social sensitivity such as virtual assistants and social robots.

The future development directions of Social-MMU include:

• Dynamic Interaction Evaluation: Expand to multi-turn dynamic interactions to simulate real social dialogues;
• Cross-modal Expansion: Integrate audio (tone, speech rate) and temporal information to enhance evaluation comprehensiveness;
• Enhanced Cultural Adaptability: Further expand coverage of cross-cultural scenarios to improve global universality;
• Integration with Downstream Tasks: Explore the correlation between social intelligence evaluation results and actual application performance.

Social-MMU marks an important expansion of multimodal AI evaluation from basic perceptual capabilities to advanced social cognitive capabilities. As AI integrates into human society, evaluating and improving its social intelligence becomes increasingly critical. The open-source nature of this project will promote the community to jointly improve the framework and help multimodal AI make continuous progress in the field of social intelligence.