MeMo: A New Paradigm for Knowledge Enhancement of Models Using Memory

MeMo is a modular knowledge enhancement framework whose core lies in encoding new knowledge through an independent memory model. It can capture cross-document relationships, resist retrieval noise, and avoid catastrophic forgetting without modifying LLM parameters. It supports plug-and-play integration into both open-source and closed-source LLMs, providing a new paradigm to solve the problem of knowledge update after LLM deployment.

Large language models (LLMs) have frozen parameters after pre-training and cannot automatically absorb new knowledge, which restricts real-time information applications. Traditional solutions have limitations:

• Retrieval-Augmented Generation (RAG) : It is difficult to capture complex cross-document relationships, susceptible to retrieval noise, and retrieval costs grow linearly with the size of the knowledge base;
• Parameter Update (Fine-tuning/Continual Learning) : It leads to catastrophic forgetting, and closed-source LLMs cannot access weights, making this solution infeasible.

MeMo proposes a new paradigm of 'Memory as Model', decoupling knowledge storage from language generation:

1. Modularity and Separability: The memory model independently encodes new knowledge, while the LLM focuses on generation; both can be optimized independently;
2. Cross-Document Relationship Modeling: Captures cross-document patterns such as entity coreference and causal chains through a dedicated encoder;
3. Retrieval Noise Robustness: End-to-end training enables the memory model to extract useful information from noise;
4. Inference Efficiency: Retrieval cost is independent of the size of the knowledge base, supporting low-latency queries of massive knowledge.

MeMo consists of three core components:

• Encoder: Uses a hierarchical attention mechanism to capture the internal semantic structure of documents and inter-document association patterns, converting new documents into compact vectors;
• Memory Storage: A structured key-value pair network where keys correspond to query semantic features and values store knowledge content, supporting efficient updates and retrieval;
• Retriever: A learning-based matching function that understands query intent and matches the most relevant memory entries.

The research team verified MeMo's performance on three benchmark tests:

• BrowseComp-Plus: Demonstrates the ability to handle large-scale unstructured data;
• NarrativeQA: Verifies cross-document narrative understanding and relationship tracking capabilities;
• MuSiQue: Achieves significant performance improvement in multi-hop question answering tasks. The results show that MeMo outperforms RAG and parameter update methods, with no catastrophic forgetting.

MeMo's modular design has broad application prospects:

• Enterprise Knowledge Management: Dynamically updates the knowledge base without model retraining;
• News Media: Tracks events in real time and establishes correlation analysis;
• Academic Research: Tracks the latest papers and builds domain knowledge graphs. Its compatibility with both open-source and closed-source LLMs reduces the application threshold of advanced AI technologies and provides a general framework for knowledge enhancement.