S-Bench: A Benchmark for Evaluating Social Intelligence of Multimodal Large Language Models

S-Bench is the first comprehensive benchmark suite dedicated to evaluating the social intelligence capabilities of multimodal large language models. Addressing the limitations of existing evaluations, it covers dimensions such as theory of mind, emotion recognition, and social norms, using multimodal inputs and multi-dimensional evaluation metrics. It provides a standardized tool for model development, product selection, and academic research, while promoting future directions like cross-cultural expansion and dynamic interactive evaluation through the open-source community.

Limitations of Existing Evaluations

Traditional LLM evaluations focus on knowledge reserve (e.g., MMLU), reasoning ability (e.g., GSM8K), and language skills, but fail to assess performance in real social scenarios (such as understanding sarcasm or microexpressions).

Necessity of Multimodality

Social interaction is multimodal (language + facial expressions + body language + tone), so evaluating social intelligence requires simultaneous processing of text, images, videos, and other information.

Evaluation Dimensions

1. Theory of Mind: Inferring intentions, false beliefs, decision differences
2. Emotion Recognition: Facial/voice/text emotions, complex emotional states
3. Social Norms: Appropriate behavior, cultural etiquette, consequences of violations
4. Interpersonal Reasoning: Relationship types, power structures, social strategies
5. Moral Judgment: Dilemma analysis, fairness, cross-cultural differences

Dataset Construction

Diversity (age/gender/culture), authenticity, difficulty gradient, anti-contamination

Technical Implementation

• Multimodal inputs: Image-text, video, plain text, audio
• Evaluation metrics: Accuracy, consistency, interpretability, human alignment

1. Multimodal Fusion Challenge: Single-modal performance is good, but performance drops significantly in multimodal integrated tasks
2. Cultural Bias Exposure: More familiar with Western cultural norms, insufficient understanding of other cultures
3. Superficial Emotion Understanding: Can recognize obvious emotions, but has limited understanding of subtle/contradictory/repressed emotions

1. Model Development Guidance: Optimize weak links through fine-grained results
2. Product Selection Reference: Provide a basis for model comparison for applications like virtual assistants and social robots
3. Academic Research Platform: Standardized evaluation tools promote progress in the field of social intelligence

1. Dynamic interactive evaluation: Simulate real-time social dialogue
2. Embodied intelligence expansion: Evaluate capabilities in physical social scenarios
3. Cross-cultural deepening: Strengthen evaluation of non-Western cultural norms
4. Long-term social memory: Evaluate the model's ability to maintain long-term social memory

S-Bench adopts an open-source model and encourages community participation:

• Dataset expansion: Accept new test scenarios
• Evaluation method improvement: Optimize metrics and processes
• Cross-cultural contributions: Solicit test cases from different cultures
• Model submission: Support applications for evaluating new models