MemTrace: An Error Tracing and Attribution Framework for LLM Memory Systems

MemTrace is an open-source framework developed by the NLP Lab at Zhejiang University. Using execution graph tracing technology, it helps developers locate the root causes of errors in LLM memory systems and provides automatic diagnostic reports and optimization solutions. This framework supports debugging of mainstream memory systems such as RAG, Mem0, and EverMemOS, addressing the pain point that traditional logs cannot answer "why an error occurred" and improving development and maintenance efficiency.

LLM memory systems are core components of long-term dialogue agents (e.g., RAG, Mem0, EverMemOS), but their complexity makes debugging difficult. The root causes of errors may lie in fact extraction, memory update, retrieval, or generation stages. Traditional logs only record what happened but cannot explain the reasons, and manual debugging is inefficient—especially infeasible in production environments.

MemTrace abstracts the execution process of memory systems into an "operation-variable" execution graph (variables include user messages, memory fragments; operations include fact extraction, retrieval). Its core capabilities include: 1. Reverse tracing: trace back to faulty operations from the incorrect answer; 2. Attribution analysis: locate faults and predict error types (fact omission, memory overwriting, etc.).

MemTrace consists of four main components: 1. smartcomment: a lightweight Python tool that records execution graphs via decorators/context (already integrated with MemBase); 2. MemTraceBench: a benchmark dataset containing failure cases and annotations for Long-Context, RAG, Mem0, and EverMemOS; 3. GraphTraceAgent: an intelligent agent based on AgentScope that interactively explores execution graphs to locate faults; 4. Diagnostic report and automatic optimization: summarize attribution results to generate reports and support automatic prompt rewriting for optimization.

MemTrace is applicable to: 1. Development debugging: quickly identify design flaws (e.g., missing information in prompts); 2. Production troubleshooting: automatically analyze online faults; 3. Iterative optimization: quantify error distribution for targeted improvements; 4. Academic research: MemTraceBench provides a standardized evaluation benchmark.

MemTrace requires Python 3.12 or higher and can be installed via pip/uv. It needs an OpenAI-compatible API and an embedding model (e.g., Qwen3-Embedding-4B). Integration only requires adding smartcomment annotations at key operation points, and MemBase provides examples. It also supports a Streamlit annotation interface for visualizing execution graphs and annotated cases.

Current limitations: only supports text memory systems, with limited multi-modal tracing; attribution accuracy depends on the capabilities of the underlying LLM, and complex execution graphs require larger context. Future directions: expand multi-modal support, optimize large-scale execution graph processing, and refine attribution granularity (e.g., prompt fragments).

MemTrace is an important advancement in LLM observability, introducing software engineering debugging techniques into AI development and providing transparency and controllability for memory systems. For developers, it is not just a debugging tool but also a systematic way of thinking, helping to transform black-box memory systems into traceable and optimizable white-box processes.