MemBoost: A Memory-Enhanced Framework for Cost-Aware LLM Inference

MemBoost is a memory-enhanced framework for cost-aware LLM inference, core using the "Retrieve-or-Upgrade" paradigm. It works collaboratively through three key components: Associative Memory Engine (AME), Large Model Oracle, and Meta Controller (MC), significantly reducing costs while maintaining the quality of large model inference. This framework addresses the redundant computation problem caused by a large number of repeated queries in production environments, enabling historical answer reuse and intelligent routing, and provides a practical cost optimization path for LLM service providers.

Large Language Models (LLMs) have high deployment costs, requiring multiple high-end GPUs for inference, especially in long-text reasoning scenarios. In production environments, there are a large number of repeated or similar queries leading to redundant computation. However, traditional Retrieval-Augmented Generation (RAG) methods focus on knowledge grounding and do not address the repeated query problem in interactive services, leaving room for cost optimization.

The core of MemBoost is the "Retrieve-or-Upgrade" paradigm, with three components collaborating to balance quality and cost:

1. Associative Memory Engine (AME)：Stores auxiliary knowledge and historical query-answer pairs, supports fast semantic retrieval and write-back, and accumulates reusable content as the service runs;
2. Large Model Oracle：A high-capability model used as a backup, only called when retrieval information is insufficient, ensuring answer quality does not fall below the baseline;
3. Meta Controller (MC)：Implemented with a lightweight LLM, responsible for routing decisions (reuse memory or upgrade to Oracle) and write-back decisions (judge whether to store new answers), reducing expensive calls with low overhead.

The research team used Zipf distribution to simulate real repeated query patterns (α=0.8/1.1/1.4) on the MMLU-Pro dataset, testing the comparison between different lightweight MC models (Qwen-3.5-2B, Ministral-3-3B, Qwen3-4B) and the Oracle (Qwen3-14B):

Method	Zipf 0.8	Zipf 1.1	Zipf 1.4
Oracle (Qwen3-14B)	76.4%	79.9%	85.0%
MemBoost (Qwen3.5-2B)	76.7%	81.8%	87.4%
MemBoost (Ministral-3B)	76.2%	79.7%	85.0%
MemBoost (Qwen3-4B)	76.1%	79.8%	85.0%

Key Findings:

• Quality Maintenance/Exceedance: When Qwen3.5-2B is used as MC, accuracy is comparable to or even higher than Oracle (e.g., 87.4% at Zipf 1.4);
• Higher Repetition Rate Brings Greater Benefits: Accuracy increases significantly as α rises;
• Lightweight MC is Sufficiently Intelligent: Small models can make effective decisions, verifying the concept of "small models directing large models".

Core Differences Between MemBoost and Standard RAG:

• Answer Reuse vs. Knowledge Grounding: Directly reuse complete historical answers instead of only retrieving documents;
• Continuous Learning: The memory bank grows with the service, forming a positive cycle;
• Interactive Design: Adapts to repeated patterns in multi-turn/multi-user scenarios;
• Cost-Aware Routing: Explicitly balances quality and cost.

Application Significance:

• Progressive Deployment: No need to replace existing infrastructure;
• Immediate Benefits: Starts accumulating memory upon launch, with cost-effectiveness continuously improving;
• Quality Assurance: Oracle as a backup avoids quality sacrifice;
• Flexible Configuration: Routing strategies can be adjusted according to business needs.

Current MemBoost is based on semantic similarity retrieval, which is insufficient for answer reuse in complex reasoning chains/multi-step derivation problems. Future directions include:

• More refined memory organization methods such as hierarchical memory structures;
• Support for partial answer reuse and combinatorial reasoning;
• Domain-specific memory optimization strategies;
• Adaptive mechanisms for dynamically adjusting routing thresholds.