CRoM: Efficient Reordering and Retrieval Techniques to Mitigate Context Decay in Large Language Models

The CRoM (Context Rot Mitigation) project focuses on the context decay problem in long text processing of large language models. Through technologies such as efficient reordering, retrieval strategy optimization, and context compression, it improves the model's utilization efficiency of early information, optimizes the context management effect of RAG systems, and solves the problems of information forgetting and noise flooding.

Large language models have a context decay phenomenon when processing long texts: as the input sequence grows, attention to early information weakens, leading to the forgetting of key details. Its root causes lie in the Transformer self-attention mechanism (attention weights are scattered as the sequence grows) and the limitations of positional encoding. In RAG systems, the simple concatenation of a large number of retrieved documents has poor effect, requiring intelligent context management strategies.

The core of CRoM is dynamic selection and organization of context: to address the deficiency of traditional RAG vector retrieval which only performs overall matching, a reordering mechanism is introduced to finely evaluate relevance; at the same time, dynamic context compression technology is adopted to extract key information and reduce the number of tokens, adapting to the limited context window of LLM.

Reordering is a key link in CRoM. It uses cross-encoders to capture fine-grained interactions between queries and documents (but with high computational cost); balances recall rate and overhead through a cascading strategy (first vector retrieval of candidate sets, then cross-encoder reordering); and also explores LLM-based reordering (using semantic understanding without special training, but with high inference cost).

CRoM optimizes retrieval strategies: dense retrieval (dense vectors capture semantic relevance), sparse retrieval (such as BM25 exact matching), and hybrid retrieval; multi-vector retrieval (generating multiple vectors for different parts of the document to finely locate information); query expansion (enriching semantics) and rewriting (converting to a form suitable for retrieval) to improve effectiveness.

Context compression technologies include: extractive compression (selecting key sentences/paragraphs to retain accuracy), generative compression (LLM summarization but with the risk of information loss), and structured representation (converting to tables/knowledge graphs to improve information density); in practice, extractive and generative strategies are often combined.

CRoM is applied in scenarios such as customer service (accurate reference to product terms), legal and medical fields (professional knowledge retrieval); evaluation indicators include recall rate (finding all relevant information), precision rate (proportion of relevant information), answer quality (accuracy and completeness), and efficiency (latency, throughput); end-to-end evaluation is performed using datasets like Natural Questions.

Future directions include adaptive context windows (dynamically adjusting the range), hierarchical attention (imitating human reading strategies), multimodal context management (processing heterogeneous information such as images/audio), personalized context (combining user preferences), long-term memory mechanisms (accumulating understanding across sessions), etc., to promote the deepening of AI applications.