DeepRefine: A New Method for Optimizing Agent Knowledge Bases via Reinforcement Learning

This article introduces DeepRefine—a reinforcement learning-based automatic refinement framework for agent knowledge bases. Addressing the three major defects of existing knowledge bases (incompleteness, inaccuracy, and redundancy), DeepRefine achieves incremental optimization through multi-round interactive exploration, abductive diagnosis for defect localization, and targeted refinement actions. Its innovative Gain-Beyond-Draft (GBD) reward mechanism solves the unsupervised training problem. Experiments show that this framework can significantly improve retrieval accuracy and downstream task performance, providing a new path for dynamic knowledge base optimization. Paper link: http://arxiv.org/abs/2605.10488v1

In LLM agent applications, external knowledge bases are crucial, but their expanding scale and frequent use lead to prominent quality issues:

1. Incompleteness: Missing key evidence or broken cross-document links;
2. Inaccuracy: Low-confidence or imprecise claims;
3. Redundancy: Ambiguous expressions and coreference resolution problems. These defects accumulate to impair retrieval accuracy and downstream task performance.

DeepRefine's workflow consists of three steps:

1. Multi-round interactive knowledge exploration: Continuously converse with the knowledge base and adjust exploration strategies based on previous results;
2. Abductive diagnosis and defect localization: Identify knowledge entries that need correction, supplementation, or deletion through advanced reasoning techniques that infer causes;
3. Targeted refinement action execution: Perform incremental update actions such as adding links, correcting claims, and eliminating redundancy in a targeted manner.

Since knowledge base refinement lacks standard reference answers, DeepRefine adopts the Gain-Beyond-Draft (GBD) reward mechanism:

• Core: Measure the difference in downstream task performance before and after refinement, and give positive rewards only when performance improves;
• Training: Use policy gradient methods to adjust reasoning strategies based on GBD signals and learn optimal interaction and refinement sequences.

Experiments validated the effectiveness across multiple knowledge-intensive tasks:

• Retrieval accuracy: After eliminating errors and redundancy, retrieval accuracy improved significantly;
• Downstream tasks: Performance in question answering, summary generation, reasoning chain construction, etc., outperformed the original knowledge base model;
• Generalization ability: Maintained stable performance improvements in cross-domain scenarios.

Significance of DeepRefine:

• Automated solution: Reduces reliance on manual review;
• Scalability: Facilitates integration of new actions and reward signals;
• Direction-setting: Promotes the shift from static construction to dynamic optimization of knowledge bases, and is expected to become a standard configuration for future agent knowledge bases. Conclusion: DeepRefine combines reinforcement learning and abductive reasoning to open a new path for knowledge base quality optimization, and will support agents to provide better services in the future.