SMMU: A Benchmark Framework for Social Intelligence of Multimodal Large Language Models

SMMU is a benchmark project dedicated to evaluating the social intelligence capabilities of multimodal large language models, filling the gap in current LLM benchmarks that focus on cognitive abilities but lack systematic assessment of social intelligence. It tests models' core social intelligence capabilities such as Theory of Mind, social reasoning, and situational understanding through interactive scenarios. Adopting an open-source model to support community contributions, it provides a standardized evaluation tool for the development of social intelligence in multimodal models.

Current large language model (LLM) benchmarks mostly focus on cognitive abilities such as language understanding, mathematical reasoning, or code generation, but lack systematic evaluation of models' performance in social contexts—namely, 'social intelligence'. Social intelligence is the core of human intelligence, encompassing the ability to understand others' intentions, emotions, beliefs, and to respond appropriately in social scenarios. The SMMU project aims to fill this gap and provide a comprehensive social intelligence evaluation framework for multimodal large language models.

Social intelligence refers to an individual's ability to understand and manage social relationships, infer others' mental states, and respond appropriately in social contexts. Its key components include:

• Theory of Mind: Understanding that others have beliefs, desires, and intentions different from one's own
• Emotion Recognition: Identifying emotional states from language, expressions, or contexts
• Social Reasoning: Predicting behaviors or outcomes based on social cues
• Situational Awareness: Understanding social norms and context-dependent behaviors For multimodal models, social intelligence also requires integrating visual information (e.g., facial expressions, body language) with textual information to form a complete understanding of social scenarios.

Evaluation Dimensions

SMMU evaluates models' social intelligence from the following dimensions:

1. Belief Inference: Testing the ability to understand others' false beliefs
2. Emotion Understanding: Assessing the ability to identify and explain emotional states
3. Intention Recognition: Inferring others' intentions from behaviors or dialogues
4. Social Norm Compliance: Testing understanding of social rules and etiquette
5. Multimodal Integration: Conducting social reasoning by combining visual and textual information

Scenario Design Principles

• Naturalness: Scenarios are derived from real social interaction contexts
• Progressive Difficulty: From simple emotion recognition to complex multi-turn social reasoning
• Multimodal Fusion: Combining image, video, and textual information
• Cultural Neutrality: Avoiding biases from specific cultural backgrounds

SMMU adopts an interactive evaluation method, which is closer to real social interactions and accurately assesses models' social reasoning abilities.

The SMMU project is built using JavaScript and includes an interactive web interface (deployed via GitHub Pages). The project structure is as follows:

• Frontend Interface: Provides intuitive display of test scenarios and interaction
• Evaluation Logic: Built-in scoring mechanism to automatically evaluate model responses
• Dataset: Predefined social scenarios and expected response standards

Online access to the project: https://smmu-team.github.io/SMMU/, researchers can directly experience the benchmark test in their browsers.

Practical Application Scenarios

Social intelligence evaluation is crucial for the following applications:

• Dialogue Assistants: Understanding users' emotions and providing empathetic responses
• Educational Tutoring: Adapting to learners' emotional and cognitive states
• Mental Health Support: Identifying signs of emotional distress in users
• Customer Service: Understanding customers' emotions and intentions, and providing appropriate responses
• Social Robots: Engaging in natural social interactions with humans

Model Development Directions

Through the SMMU benchmark, researchers can:

• Identify the shortcomings of current models in social reasoning
• Guide model training to targetedly improve social intelligence
• Compare the social ability performance of different models
• Track the progress of models' social intelligence

SMMU adopts an open-source model and welcomes community contributions:

• Use the existing benchmark to test your own models
• Submit new social scenarios to expand the test set
• Improve evaluation metrics and scoring mechanisms
• Share test results and findings

Original author/maintainer: SMMU-Team, Source platform: GitHub, Original link: https://github.com/SMMU-Team/SMMU, Release date: May 24, 2026.

SMMU represents an important direction in the AI evaluation field, expanding from pure cognitive ability testing to social intelligence evaluation. As AI systems increasingly participate in human social interactions, evaluating and improving their social understanding capabilities is crucial. This benchmark not only provides researchers with a standardized evaluation tool but also points the way for the future development of social intelligence in multimodal models. We look forward to more models making progress on this benchmark, ultimately achieving AI systems with true social understanding capabilities.