UnifiedMemBench: A Unified Memory Evaluation Benchmark for Large Language Models

UnifiedMemBench is an open-source benchmark framework developed by the AceLi12138 team, aiming to systematically evaluate the memory capabilities of large language models (LLMs). It fills the gap in existing benchmarks that only focus on a single type of memory. Through three core dimensions—contextual memory, parametric memory, and retention memory—it comprehensively reveals the performance of models in different memory scenarios, providing an important tool for LLM research and applications.

Large language models are developing rapidly, but existing memory evaluation benchmarks have a problem of singularity and cannot systematically assess the comprehensive memory capabilities of models. In practical applications, LLMs need to handle immediate context, long-term parametric knowledge, and cross-session information retention simultaneously. Therefore, a comprehensive evaluation tool is urgently needed.

UnifiedMemBench divides memory evaluation into three dimensions:

1. Contextual Memory: Evaluate the model's ability to retain and utilize immediate information, such as long-text reference and dialogue state tracking;
2. Parametric Memory: Test world knowledge encoded during pre-training, including common sense and professional knowledge;
3. Retention Memory: Innovatively assess the ability to retain and update information across interactions, simulating long-term user interaction scenarios.

UnifiedMemBench adopts a modular architecture that supports flexible configuration of test scenarios. Each memory dimension is equipped with a dedicated dataset and metrics. An event-driven test case generation mechanism ensures that the evaluation is close to practical applications. Evaluation results are presented as standardized scores, facilitating horizontal comparison of models.

For researchers, this benchmark provides a systematic memory analysis tool to help identify the impact of architecture and training methods on memory. For developers, it can assist in selecting models suitable for vertical fields (such as customer service, education, and healthcare), especially in scenarios requiring long-term interaction.

UnifiedMemBench fills the gap in the comprehensive evaluation of LLM memory. The three-dimensional framework provides a tool for improving memory mechanisms. As multi-turn dialogue and personalized applications become more popular, retention memory capabilities will receive more attention. The open-source nature promotes community collaboration and benchmark improvement.