MGRAG: A Graph-Structured Multimodal Retrieval-Augmented Generation System

MGRAG is an open-source framework that combines knowledge graphs with multimodal retrieval-augmented generation. It organizes cross-modal information via graph structures to enhance the accuracy and interpretability of large language models (LLMs) in multimodal question-answering tasks.

With the continuous improvement of large language model (LLM) capabilities, retrieval-augmented generation (RAG) has become a mainstream solution to address model hallucinations and knowledge timeliness issues. However, traditional RAG systems are mainly designed for text modalities and have obvious limitations when dealing with multimodal content such as images and videos. Multimodal question-answering tasks require models to not only understand visual information but also effectively associate it with textual knowledge, which places higher demands on the design of retrieval systems.

MGRAG (Graph-based Multimodal Retrieval-augmented Generation) is an innovative framework born in this context. It organizes multimodal information by introducing graph structures, enabling unified representation and efficient retrieval of cross-modal knowledge.

The core design concept of MGRAG is to combine the structural advantages of knowledge graphs with the flexibility of multimodal retrieval. The system mainly includes the following key components:

The system uses a vision-language model (VLM) as the foundation for image understanding, providing efficient visual feature extraction through the vLLM service. Image captions are precomputed and stored for subsequent graph construction and retrieval.

MGRAG constructs document and image information into a heterogeneous graph structure. Nodes in the graph can represent text fragments, image entities, or concepts, while edges represent semantic relationships between them. This structured representation enables cross-modal reasoning.

Unlike traditional vector retrieval, MGRAG uses graph traversal for information retrieval. The system supports multiple graph retrieval strategies, including path-based retrieval, subgraph sampling, and parallel retrieval. Users can control the granularity and scope of retrieval through parameters such as path_recent_nodes.

The system integrates multiple reasoning modes, supporting direct retrieval and iterative graph expansion. The stop_detect mechanism can intelligently determine when to terminate retrieval, controlling computational overhead while ensuring recall rate.

MGRAG is developed based on Python 3.10.16, with core dependencies including:

• vLLM: Provides high-performance vision-language model inference services
• LMCache: A caching system for accelerating LLM inference
• Graph Database: Supports complex graph traversal and query operations