DKMD: A New Paradigm of Dual Knowledge-Enhanced Multimodal Dialogue System

This article provides an in-depth interpretation of the TOIS 2024 paper DKMD (Dual Knowledge-enhanced Multimodal Dialog). This framework addresses the issues of LLM hallucinations and knowledge timeliness in multimodal dialogue systems by integrating external explicit knowledge and internal implicit model knowledge, and provides an open-source implementation, offering an innovative solution for domain research and practice.

Multimodal dialogue needs to process text, visual, and other information, but there is a fundamental tension: LLMs contain massive parameterized knowledge but are static, while external knowledge is dynamic and accurate but requires integration. How to coordinate these two types of knowledge has become a core design challenge, and DKMD provides a solution for this.

Core Idea

DKMD was developed by iLearn Lab, with the core being a dual knowledge enhancement mechanism: simultaneously utilizing external knowledge bases (explicit) and internal model knowledge (implicit), complementing each other through fusion strategies.

Key Modules

• Multimodal encoder: unifies text/visual semantic representations
• Dual knowledge retrieval: external knowledge base (RAG) + internal knowledge (prompt activation)
• Knowledge fusion: layered hybrid strategy (light injection at encoding layer + dynamic selection at decoding layer)
• Response generation: generates natural responses based on fused knowledge

Enhancement Mechanisms

• Explicit: visual perception retrieval, multi-source integration, dynamic selection
• Implicit: chain-of-thought prompting, multi-step reasoning, conflict detection

Evaluation Setup

Datasets: VQAv2, VisDial, FVQA, etc.; Metrics include accuracy, knowledge correctness, fluency, etc.

Key Results

• Knowledge accuracy improved by 15-20%, alleviating hallucinations
• Visual question answering outperforms text retrieval baselines
• Response fluency did not decrease, and topic materials are more abundant
• Ablation experiments verify the necessity of dual knowledge enhancement

Open-Source Resources

Provides PyTorch implementation, training scripts, data pipelines, and pre-trained models, supporting reproduction and downstream tasks.

Application Extensions

• Domain adaptation: replacing the knowledge base can be used in vertical fields such as healthcare/law
• Multilingual support: just replace the base model and knowledge base
• Real-time information access: natively supports real-time knowledge sources

Contributions

• Theory: systematically studies the problem of multimodal knowledge fusion
• Technology: open-source baseline lowers the threshold for research
• Practice: provides references for the industry

Limitations

• Retrieval latency limits real-time performance
• Conflict handling mechanism is simple
• Long dialogue context management needs optimization

Future Directions

More intelligent retrieval strategies, end-to-end knowledge-generation joint optimization, scenario-specific optimization