ASEM: Enabling Self-Evolving Memory Systems for Large Language Model Agents

ASEM (Agentic Self-Evolving Memory) is a five-stage memory framework designed to address the issues of knowledge freezing and inability to continuously learn in Large Language Model (LLM) agents. Through structured memory organization, reinforcement learning-driven management, and value-aware retrieval, ASEM allows agents to maintain a living knowledge network across conversations and achieve self-evolution. Its core innovations include multi-attribute atomic notes, a memory manager trained with GRPO, two-stage hybrid retrieval, etc., providing a new path for the practical deployment of LLM agents and research on continuous learning.

Current LLM agents face the problem of knowledge being frozen in pre-trained parameters, with high fine-tuning costs that are difficult to execute frequently, and a lack of cross-conversation memory and experiential learning mechanisms. ASEM proposes keeping the underlying model frozen and achieving continuous learning and adaptation through an external memory bank and utility estimation to address this fundamental challenge.

ASEM's core is a five-stage memory framework:

1. Multi-attribute Atomic Notes: Include original content, embedding vectors, keywords, and other metadata, supporting multi-dimensional retrieval;
2. Reinforcement Learning-Driven Management: A memory manager trained with GRPO optimizes memory writing/organization/update strategies;
3. Two-Stage Hybrid Retrieval: First recall semantically similar content, then perform value-aware reordering (evaluating the memory's help for the task);
4. Non-parametric Utility Update: Use EMA to dynamically adjust memory utility, which is lightweight and effective;
5. Plug-and-Play Inference Backend: Supports HuggingFace and LangChain; training uses HuggingFace exclusively.

ASEM codebase core modules:

• asem/: Core functions such as memory management, retrieval, and utility estimation;
• training/: GRPO training loop (training for memory manager and response agent);
• eval/: Evaluation framework and baseline comparison;
• configs/: Default hyperparameter configurations;
• data/: Prompt words and benchmark test resources;
• scripts/: Model download and performance analysis tools. A complete training and evaluation workflow is provided, supporting benchmark testing, result generation, and manual evaluation.

Enterprise applications: Agents remember user preferences/business rules/interaction history to provide personalized services (e.g., customer service remembering customer issues, programming assistants matching code styles); Research fields: Demonstrates the application of reinforcement learning in memory management, provides new ideas for continuous/lifelong learning, and non-parametric utility updates serve as a reference for lightweight design.

Limitations: Training the memory manager requires large amounts of data and computing resources, and the accuracy of utility estimation depends on task types; Future directions: Integrate advanced RAG technology, explore memory compression and summarization, and expand to multi-modal memory (images, audio, etc.).

ASEM lays the foundation for the self-evolution of LLM agents through structured memory organization, reinforcement learning management, and value-aware retrieval. As LLM applications deepen, such memory frameworks will become key components in building intelligent and adaptive AI systems.