UnifiedMemBench: A Comprehensive Memory Evaluation Benchmark for Large Language Models

This article introduces UnifiedMemBench, an open-source evaluation framework focused on assessing the memory capabilities of large language models (LLMs). It covers three core dimensions: contextual memory, parameterized knowledge, and long-term retention, and uses an event-centric evaluation method to provide a systematic tool for evaluating LLM memory capabilities.

Large language models are developing rapidly, but traditional evaluation benchmarks lack systematic assessment of memory capabilities. Memory capability is crucial for the practicality of AI systems (e.g., coherence in multi-turn dialogues, long-term task execution). Thus, UnifiedMemBench was developed to provide a unified event-centric framework for evaluating the three memory dimensions.

Contextual Memory

Similar to human working memory, it refers to the ability to use previous information when processing current dialogues/texts, which affects the coherence of dialogues in products like customer service robots.

Parameterized Knowledge

Factual knowledge encoded into model parameters during the pre-training phase, which determines the reliability of the model as a knowledge tool.

Long-term Retention

The ability to recall specific information after a long time span, which is key for personalized AI assistants.

UnifiedMemBench uses an event-centric evaluation method, which differs from traditional static question-answering/reading tasks. It simulates real information flow by constructing time-series event scenarios, thereby improving ecological validity (the evaluation results are more relevant to practical applications).

This benchmark helps researchers identify the memory shortcomings of models and track changes in memory capabilities during iterations. It also provides a basis for developers to select appropriate models based on application scenarios (e.g., customer service requires contextual memory, knowledge Q&A requires parameterized knowledge).

As an open-source project, UnifiedMemBench provides code and datasets, supports adding new scenarios, customizing tests, and comparing model performance, ensuring that the framework evolves continuously with the development of LLM technology.

Memory capability is key to measuring the practicality of LLMs. Through its three-dimensional framework and event-centric method, UnifiedMemBench provides the community with a systematic evaluation tool, which will promote the improvement of user experience for AI systems.