SocialMemBench: Capability Gaps and a New Evaluation Benchmark for AI Memory Systems in Social Group Scenarios

SocialMemBench is the first evaluation benchmark for AI memory systems designed for multi-person social group scenarios, revealing significant deficiencies in current mainstream memory frameworks when handling group norms, cross-person knowledge, and dynamic changes in group members. This article will discuss aspects such as background, framework design, evaluation results, and future directions.

Current mainstream AI assistant memory systems are designed based on single-user dialogues, performing well in one-on-one scenarios but showing flaws when deployed in multi-person social group scenarios. Group scenarios require AI to anchor shared history, distinguish group norms from individual exceptions, handle dynamic changes in members, and manage cross-person knowledge. Existing evaluation benchmarks focus on binary dialogues (e.g., MultiWOZ) or workplace dialogues (e.g., MeetingBank), and do not cover the informal, relationship-oriented characteristics of social groups, leading to a lack of academic evaluation support for industrial deployment.

SocialMemBench's core design features include:

1. Group Prototype Diversity: Covers five types: close friend circles, family groups, interest communities, acquaintance networks, and entertainment groups;
2. Scale Stratification: Covers small (4-6 people), medium (7-15 people), and large (16-30 people) groups;
3. Data Scale: Contains 43 synthetic social networks, 430 character profiles, 7355 dialogue turns, and 1031 Q&A pairs, all manually verified to ensure authenticity and answerability.

Evaluation questions are divided into nine categories: fact recall, time reasoning, member identification, relationship reasoning, norm understanding, exception handling, dynamic attribution, cross-person knowledge, and conflict resolution. Five failure modes include:

• Single-stream confusion: Confusing multiple dialogue streams;
• Tense coverage: New information overwriting old information;
• Scaled entity merging: Incorrectly merging people/concepts;
• Cross-person knowledge deficiency: Unable to integrate multi-member information;
• Norm-individual confusion: Confusing group norms with individual exceptions.

Mainstream open-source memory frameworks (Mem0, LangMem, Graphiti, Cognee) have weighted scores of only 0.12-0.18, far lower than the naive retrieval baseline (0.345) and full context baseline (0.369). Gemini 2.5 Flash in the full context setting scored 0.721, still lower than the blind evaluation average of 0.98, indicating that the problem stems from the complexity of information organization, relationship reasoning, and group cognition.

SocialMemBench brings multiple insights:

1. Architecture Reflection: Existing architectures are unsuitable for group scenarios; they need to support individual/group perspectives, dynamic relationships, and social reasoning;
2. Evaluation Upgrade: Need fine-grained evaluation frameworks (e.g., nine question categories + failure modes);
3. Human-Computer Interaction: Need to consider social science issues such as memory presentation methods and privacy boundaries. The benchmark code and dataset have been made public, which will promote the technological progress of memory systems.