SMMU: A New Benchmark for Evaluating Social Intelligence of Multimodal Large Language Models

SMMU is an open-source evaluation framework dedicated to assessing the social intelligence of multimodal large language models. It aims to fill the gap in existing benchmarks that lack systematic evaluation of social understanding capabilities. Through multi-dimensional assessment, real-scenario orientation, and multi-modal input fusion design, it provides researchers with standardized tools to promote domain development and offer references for application selection.

With the breakthroughs of multimodal large language models in visual and text tasks, the community has turned its attention to their social intelligence performance. Social intelligence is the core of human intelligence, but existing benchmarks focus on perceptual and cognitive tasks and lack systematic evaluation. The SMMU project was born to fill this gap.

Social intelligence evaluation differs from traditional tasks; it requires models to understand interpersonal relationships, emotional states, social norms, etc. For example, in a party photo scenario, the model needs to identify relationships, understand emotions, infer intentions, and judge the compliance of behaviors, which places higher demands on multi-modal fusion and commonsense reasoning.

SMMU follows three core principles: 1. Multi-dimensional assessment: covering dimensions such as emotion recognition, social relationships, situational reasoning, and cultural sensitivity; 2. Real-scenario orientation: data comes from real scenarios like family gatherings and workplaces; 3. Multi-modal input fusion: samples include images and text to simulate human cognitive processes.

SMMU adopts a modular architecture: the dataset construction module is responsible for data collection and annotation (with multi-round verification to ensure quality); the evaluation engine provides a standardized process (supporting batch testing and statistics); the analysis toolset includes visualization and statistical modules to help understand model performance.

SMMU brings value to the field: 1. The standardized benchmark promotes performance comparison across teams; 2. It reveals model deficiencies and points out research directions; 3. It provides references for the selection of applications such as virtual assistants and educational robots.

SMMU is released as open-source; resources can be obtained via GitHub, and community contributions are welcome. Future plans include expanding evaluation dimensions, exploring dynamic interaction evaluation, and conducting interdisciplinary collaborations with cognitive science and psychology to ensure the evaluation aligns with human social cognitive principles.