SMMU Benchmark: Evaluating Social Intelligence of Multimodal Large Language Models

This article introduces the SMMU (Social Intelligence Benchmark for Multimodal Understanding) project, a benchmark framework specifically designed to evaluate the social intelligence capabilities of multimodal large language models. While the current AI evaluation system is comprehensive, it has long overlooked social intelligence—a core capability that is crucial for AI to integrate into human society. By deconstructing social intelligence into multiple dimensions (emotion recognition, theory of mind, social context understanding, reasoning and prediction), and adopting a multimodal test design and hybrid evaluation method, SMMU fills this evaluation gap, providing model developers and researchers with diagnostic tools and a common platform to drive AI evaluation toward a direction that is closer to human real-world capabilities.

The current evaluation system for multimodal large language models covers multiple dimensions such as visual question answering and image captioning, but social intelligence—a core component of human intelligence—has been long overlooked. Social intelligence involves understanding others' emotions, intentions, and beliefs, interpreting social context clues, and predicting the trajectory of interpersonal interactions. It is crucial for AI systems that serve humans; the practical value of AI lacking social intelligence will be greatly reduced.

The SMMU project aims to fill the gap in social intelligence evaluation, with its core contribution being the definition of the problem space for social intelligence evaluation and the construction of a standardized dataset. The framework decomposes social intelligence into four sub-dimensions: basic-level emotion recognition (identifying non-verbal cues such as facial expressions and body language); deep-level theory of mind (understanding others' different beliefs and intentions); social context understanding (grasping scene norms and cultural backgrounds); and the highest-level social reasoning and prediction (predicting the development of situations and behavioral consequences).

A notable feature of SMMU is its multimodal design; test materials integrate visual (images, videos) and linguistic information, closely resembling real social scenarios. The evaluation uses a hybrid model of multiple-choice questions and open-ended answers: multiple-choice questions facilitate large-scale automated evaluation, while open-ended answers reveal details of reasoning. In addition to accuracy, metrics also focus on confidence calibration, hierarchical performance, and adversarial robustness to comprehensively reflect the behavioral characteristics of models.

Preliminary evaluations based on SMMU show that existing models perform well in basic emotion recognition tasks, but there is a significant gap between them and humans in complex theory of mind and social reasoning tasks. Failure mode analysis found that models are prone to systematic errors when integrating multiple social cues, handling long-term relationship dynamics, or dealing with culture-specific norms, which triggers reflections on training data and pre-training objectives.

The application value of SMMU includes: providing developers with diagnostic tools to help identify shortcomings in social intelligence; and establishing a fair comparison platform for researchers. Future directions can include expanding the types of social scenarios, adding cross-cultural samples, developing fine-grained error analysis tools, and exploring the relationship between social intelligence and other cognitive abilities. Social intelligence is the cornerstone of human-AI collaboration; its evaluation and cultivation should become an important part of AI development to facilitate AI safety and alignment.