ColGraphRAG: A Multimodal Question Answering System Based on Query-Conditioned Evidence Graph and ColEmbed Re-Ranking

ColGraphRAG implements the multimodal GraphRAG method from the ACL 2025 paper. It achieves end-to-end question answering on WebQA and MultiModalQA through constructing question-specific evidence graphs, ColEmbed MaxSim image re-ranking, and large language model generation, addressing core challenges such as cross-modal information association, precise evidence localization, and evidence-based answer generation.

Traditional question answering systems rely on text corpora, but real-world information is often in mixed text-image formats. Benchmarks like WebQA and MultiModalQA require handling complex combinations of images, tables, and text to answer cross-modal multi-hop questions. Core challenges include associating information across different modalities, precisely locating evidence among massive candidates, and generating evidence-based answers. ColGraphRAG addresses these challenges with a query-driven multimodal reasoning pipeline based on the ACL 2025 Findings paper, modeling evidence relationships via graph structures and achieving image re-ranking through ColEmbed's late interaction mechanism.

ColGraphRAG decomposes the question answering process into six stages:

1. Corpus Slicing and Export: Provides dedicated scripts for WebQA and MMQA, converting data into a unified JSONL format;
2. Graph Schema Generation: Uses large language models to generate question-specific graph schemas, defining entity types and relationship structures;
3. Entity and Relationship Extraction: Extracts structured graph elements from documents based on the graph schema;
4. Evidence Graph Construction: Instantiates into a graph structure using NetworkX and exports to GraphML format;
5. Core Reasoning: Converts graphs to text representations, performs ColEmbed MaxSim image re-ranking, and uses LLMs to generate answers by combining graph context and images;
6. Answer Evaluation: Uses metrics like QA-FL and QA-Acc for WebQA, calculates exact match rate and F1 score for MMQA, and supports modality-layered analysis.

ColGraphRAG uses the NVIDIA Llama-Nemotron-ColEmbed-VL-3B-v2 model for image re-ranking. Its late interaction architecture independently encodes queries and images, with fine-grained token-level interaction during similarity calculation. The MaxSim operation is core: for each token in the query, it finds the maximum similarity among image tokens and aggregates them, capturing precise correspondence between the query and local image regions. This is more discriminative than global embeddings, filtering the most relevant visual evidence.

The system uses Google Gemma-4-E4B-IT (a 4 billion-parameter multimodal instruction-tuned model) by default for answer generation. During reasoning, it receives text representations of graph structures and images re-ranked by ColEmbed. It supports a --dry-run mode for testing logic. Full operation requires approximately 16GB of VRAM, with A100/H100 GPUs recommended.

It supports two benchmarks: WebQA (multi-hop web question answering) and MMQA (joint reasoning over text/tables/images). It provides corpus preparation guidelines (official data download, directory structure, toy dataset). Experimental reproducibility is ensured through environment isolation (Python3.10+ virtual environment, requirements.txt), centralized configuration management (config/YAML), automatic model download (util scripts), and Chinese/English Jupyter Notebook tutorials.

ColGraphRAG integrates graph structure reasoning, multimodal retrieval, and generative question answering. Compared to pure-text RAG, it explicitly models evidence associations; compared to traditional multimodal models, it achieves precise image filtering via late interaction, giving it an advantage in cross-modal multi-hop reasoning problems. It provides an open-source implementation of the ACL 2025 paper as a benchmark for the research community, with a modular design allowing component replacement. Application prospects include scenarios like intelligent search, knowledge question answering, and content understanding.