MemTrace: A Framework for Error Tracing and Attribution in LLM Memory Systems

Original Author/Maintainer: Zhejiang University NLP Team (zjunlp) Source Platform: arXiv Original Title: MemTrace: Tracing and Attributing Errors in Large Language Model Memory Systems Original Link: http://arxiv.org/abs/2605.28732v1 Publication Date: 2026-05-27

The MemTrace framework converts the memory pipeline into an executable memory evolution graph to achieve fine-grained information flow tracing and automatic error attribution. It has been verified on the MemTraceBench benchmark to improve downstream task performance by 7.62%.

Memory mechanisms are crucial for long-range reasoning in LLMs. Existing memory systems (such as RAG, long context windows, Mem0, etc.) have issues of unreliability and difficulty in debugging:

• Hard to locate error sources: Developers cannot determine whether the problem lies in the retrieval, integration, or generation stage
• Lack of fine-grained visibility: The dynamic evolution process of memory systems (information synthesis, propagation, corruption) cannot be traced by existing tools

Construction of Memory Evolution Graph

Model memory operations (retrieval, storage, update, generation) as graph nodes and information dependency relationships as edges. Record state changes before and after operations through the execution graph to achieve complete tracing of information flow.

Automatic Attribution Method

When the system produces an error output:

1. Identify the subgraph of operations related to the failure
2. Analyze the contribution of each operation to the final output
3. Locate the key operations that caused the error

MemTraceBench Benchmark

Covers four representative memory systems: Long-Context (long context), RAG (Retrieval-Augmented Generation), Mem0 (dedicated memory layer), EverMemOS (OS-level memory management)

Experimental Findings

Memory failures have systematic roots: information loss, retrieval misalignment

Closed-Loop Optimization Effect

The closed-loop system based on attribution signals (detect failure → attribute → optimize prompt → verify) can improve end-to-end task performance by 7.62%

1. Memory Evolution Graph Construction: Unify and abstract different memory systems into graph representations, which requires in-depth understanding of the common abstractions of each mechanism
2. Execution Tracing: Balance information richness and performance overhead, record state changes of each operation
3. Attribution Algorithm: Combine graph algorithms and causal inference techniques to extract causal relationships from complex dependencies to locate problems

1. Memory system reliability requires systematic debugging tools to replace inefficient manual debugging
2. Different memory systems face similar challenges; unified graph representation supports cross-system comparison and transfer learning
3. Attribution signals can guide automatic repair, paving the way for self-improving memory systems

Limitations

• Only supports text memory systems; support for multimodal memory (images, videos) needs to be expanded
• Attribution accuracy depends on the quality of graph construction; complex memory mechanisms may lose key information

Future Directions

• Expand to more types of memory systems
• Improve the accuracy and efficiency of attribution algorithms
• Explore more complex automatic repair strategies
• Integrate into mainstream development toolchains