KCM: Enhancing Retrieval-Augmented Vision-Language Large Models via Knowledge Conflict Mitigation

This article is the open-source implementation of an AAAI 2026 accepted paper, proposing the Knowledge Conflict Mitigation (KCM) framework. Targeting the inconsistency between retrieved knowledge and the model's internal knowledge in Retrieval-Augmented Vision-Language Models (Retrieval-Augmented VLMs), it improves the accuracy and reliability of model responses, reduces hallucinations, and enhances system credibility by explicitly detecting, resolving, and integrating conflicting knowledge.

Retrieval-Augmented Generation (RAG) technology has been extended to vision-language models, forming Retrieval-Augmented VLMs, but there are knowledge conflict issues: manifested as factual (e.g., incorrect penguin habitat), timeliness (e.g., outdated presidential information), granularity (detailed vs. rough), and visual-text (contradiction between images and retrieved text) conflicts. Without handling these, it will lead to decreased response quality, invalid confidence, loss of user trust, and security risks.

KCM is based on three key insights: conflicts are normal, simple fusion is insufficient, and explicit modeling is needed. It follows three principles: conflict detection (calculating consistency, identifying types and severity), conflict resolution (retrieval priority/internal priority/fusion/uncertainty expression), and knowledge integration (conflict-aware attention, multi-source fusion, traceability).

1. Conflict Detection Module: Extract the model's internal response (pre-inference), obtain retrieved documents, calculate conflict scores (semantic similarity, uncertainty estimation, explicit comparison); 2. Conflict Resolution Strategies: Retrieval priority (increase retrieval weight), internal priority (supplementary retrieval), fusion (gated weighting), uncertainty expression (explicit explanation); 3. Integration Architecture: Conflict-aware attention (dynamically fuse internal and retrieved knowledge), multi-modal three-way fusion (vision + internal + retrieval), hierarchical processing (paragraph/sentence/document level).

Data Construction: Adversarial construction (generate wrong answers), timeliness construction (new and old knowledge bases), multi-source fusion (different knowledge sources); Training Objective: Total loss = generation loss + λ1 conflict detection loss + λ2 knowledge selection loss; Training Techniques: Curriculum learning (from simple to complex conflicts), contrastive learning (pull correct outputs closer, push wrong outputs away).

Evaluation Metrics: Generation quality (accuracy, completeness, fluency), conflict handling ability (detection accuracy, strategy appropriateness, traceability accuracy), system-level metrics (hallucination rate, consistency, user satisfaction); Results: Significant accuracy improvement on benchmark datasets, more obvious improvement on conflict subsets, reduced hallucination rate; Ablation experiments verify the contribution of each component, and the complete framework has the best effect; Case analysis shows the advantages in handling timeliness, visual-text conflicts, and uncertainty expression.

Application Scenarios: Real-time knowledge Q&A (news images, product recognition, landmarks), professional fields (medical imaging, legal documents, scientific literature), multi-modal dialogue systems; Limitations: High computational cost, generalization ability needs improvement, evaluation challenges; Future Directions: Efficient conflict detection, adaptive strategy learning, multi-turn dialogue processing, extension to pure text RAG, multi-modal support, real-time system optimization.

KCM brings a new perspective to Retrieval-Augmented VLMs, emphasizing the importance of explicitly handling knowledge conflicts to improve system accuracy and reliability. It is of great significance to AI safety and practicality, provides a technical route for building multi-modal RAG systems, and helps to create more robust vision-language understanding systems.