Multimodal RAG System: Intelligent Retrieval-Augmented Generation Integrating Vision and Text

Traditional Retrieval-Augmented Generation (RAG) systems often lose information about visual elements such as charts and table screenshots when processing PDF documents, while these contents are often the source of key answers. The multimodal RAG system built in this project treats images as first-class citizens equal to text. It uses CLIP and LLaVA-style models to achieve unified retrieval and generation for mixed PDF documents (research papers, slides, etc.) and scattered screenshots. Finally, it outputs grounded answers that both cite text paragraphs and label the charts they rely on, solving the pain points of traditional RAG.

In the field of information retrieval and knowledge management, RAG technology has become an important paradigm for large language model applications. However, traditional RAG systems treat PDFs as pure text blocks when processing them, leading to complete loss of visual element information. This project aims to solve this problem, realize unified processing of mixed PDF documents and scattered screenshots, and generate explainable answers with chart references.

The system adopts a multi-path recall fusion strategy, and the overall workflow is as follows:

1. Ingestion: Process PDF/image folders using pypdf+pymupdf for text chunking and image cropping;
2. Indexing: Encode text chunks with sentence-transformers and images with CLIP, then store them in a FAISS multimodal index;
3. Retrieval: Use the RRF fusion strategy, combining three retrieval modes: text-text, image-text, and text-image;
4. Generation: Generate grounded answers with source annotations through a LLaVA-style VLM.

Retrieval Modes:

1. Text-text retrieval: Encode and retrieve text chunks using sentence-transformers/all-MiniLM-L6-v2;
2. Image-text retrieval: Use the CLIP text encoder to index images, supporting natural language descriptions to find charts;
3. Text-image retrieval: When users paste an image with a question, retrieve related text to supplement information.

Generation Phase: Use the llava-hf/llava-1.5-7b-hf model, view the fused top-k text paragraphs and up to four images, generate accurate and explainable grounded answers, and label the information sources.

Basic Framework: Python3.12 + PyTorch2.6 + Transformers4.50 Multimodal Models: Open CLIP2.30 (image-text alignment), sentence-transformers3.3 (text semantic encoding) Vector Storage & Retrieval: FAISS (efficient similarity search, separate storage for multimodal indexes) Web Services: FastAPI0.116 (asynchronous API gateway), Streamlit1.40 (interactive chat interface) Document Processing: pypdf+pymupdf (PDF parsing and image extraction) Auxiliary Tools: Pydanticv2 (request validation), LangChain0.3 (optional query rewriter)

The project maintains three sets of evaluation datasets:

1. slidedecks.jsonl: Test the ability to understand commercial presentation documents;
2. papers_with_figures.jsonl: Verify the ability to associate text and charts;
3. screenshots_ui.jsonl: Evaluate UI design and user flow understanding.

Evaluation metrics include recall@5, recall@10, and MRR (Mean Reciprocal Rank), which comprehensively measure retrieval quality.

Core Value: Retain the interpretability and controllability of traditional RAG, expand application scenarios to rich document environments, and achieve more comprehensive information coverage through cross-modal retrieval. Applicable Scenarios: Academic research (processing papers with charts), business analysis (parsing financial reports), technical documents (understanding architecture diagrams), medical diagnosis (integrating imaging reports and clinical records).

Quick Start:

1. Install dependencies: pip install -r requirements.txt
2. Build index: python scripts/build_index.py --src data/raw --out data/index
3. Start API: python serve.py (access http://localhost:8000)
4. Start Streamlit interface: streamlit run streamlit_app.py

API Endpoints: /upload (file upload), /search (pure retrieval), /ask (full RAG), /health (health check)

Containerized Deployment: docker compose up --build, supporting deployment on cloud platforms and local servers.