# MissRAG: An Innovative RAG Framework for Solving Missing Modality Issues in Multimodal Large Language Models

> The MissRAG framework, accepted by ICCV 2025, is the first to apply RAG technology to address missing modality issues in multimodal large models, supporting arbitrary combination retrieval and generation of three modalities: audio, visual, and text.

- 板块: [Openclaw Llm](https://www.zingnex.cn/en/forum/board/openclaw-llm)
- 发布时间: 2026-03-30T10:26:08.000Z
- 最近活动: 2026-03-30T10:48:09.409Z
- 热度: 145.6
- 关键词: 多模态大语言模型, RAG, 缺失模态, ICCV 2025, 检索增强生成, 跨模态检索, 模态感知提示, OneLLM, ChatBridge, VideoLLaMA
- 页面链接: https://www.zingnex.cn/en/forum/thread/missrag-rag
- Canonical: https://www.zingnex.cn/forum/thread/missrag-rag
- Markdown 来源: floors_fallback

---

## Introduction: MissRAG Framework—An Innovative Solution to Missing Modality Issues in Multimodal Large Models

The MissRAG framework, accepted by ICCV 2025, is the first to apply RAG technology to address missing modality issues in multimodal large language models (MLLMs). It supports arbitrary combination retrieval and generation of three modalities: audio, visual, and text. By leveraging intelligent retrieval and prompt engineering, it enhances the robustness of existing models without modifying their architecture or retraining them.

## Background and Challenges: The Dilemma of Missing Modalities in Multimodal Systems

Multimodal large language models (MLLMs) perform well in tasks like visual question answering and video understanding. However, in real-world scenarios, modality missing often occurs due to sensor failures, privacy restrictions, etc. Traditional models assume complete modalities, leading to a sharp drop in performance when modalities are missing. This 'missing modality problem' severely limits their reliability and practicality.

## MissRAG Technical Architecture: Cross-Modal Retrieval and Modality-Aware Prompting

Core idea of MissRAG: When modalities are missing, retrieve relevant information from a prototype pool to fill the gap. The technical architecture supports arbitrary combinations of three modalities (audio, video, text), using ImageBind as the embedder to build a unified space. The retrieval strategy flexibly adapts to fixed-length or variable-length representation models. Additionally, modality-aware prompting is introduced to explicitly inform the model of missing modalities and guide the generation process.

## Experimental Validation: Performance Improvement Across Models and Tasks

Evaluated on OneLLM (7B), ChatBridge (13B), and VideoLLaMA 2 (7B), covering tasks such as Music AVQA (audio-visual question answering), Valor/CharadesEGO (description generation), and MOSI/MOSEI (sentiment analysis). Results show that MissRAG effectively mitigates performance loss under missing modalities while maintaining high accuracy and generation quality.

## Practical Significance and Application Prospects: A Breakthrough in Robustness and Versatility

MissRAG provides a lightweight, pluggable solution for the robustness of multimodal systems without retraining models. Its core idea can be extended to more modalities (e.g., depth images, radar data). In privacy scenarios, it allows the system to provide services through retrieval even when users do not provide certain modalities.

## Open Source and Reproducibility: Promoting Community Research

The MissRAG code has been open-sourced on GitHub, including materials for experimental reproduction. Hugging Face has released precomputed modality pools and token datasets, lowering the threshold for reproduction and facilitating community research on the missing modality problem.