Open Source Multimodal Multilingual RAG System: A Document Q&A Solution Supporting Offline Operation in 100+ Languages

This article introduces the open-source project Multimodal-Multilingual-RAG, which aims to address the limitations of existing RAG systems that only support English and plain text. The system has three core features:

1. Multilingual Support: Covers over 100 languages (including mixed languages like Hinglish and Manglish);
2. Multimodal Understanding: Processes text, tables, and images in PDFs;
3. Fully Offline Operation: Local deployment with zero API cost and privacy protection. The project uses a practical tech stack, is easy to deploy, and is suitable for scenarios like multilingual document processing.

Retrieval-Augmented Generation (RAG) is a mainstream architecture for large language model applications, but most open-source projects have two major limitations: they only support English and can only handle plain text content. For scenarios involving multilingual documents or PDFs with charts and images, existing solutions often fall short.

Multilingual Support

Uses the multilingual-e5-large embedding model, supporting real-world internet multilingual text and code-mixed scenarios (e.g., Hinglish, Manglish), allowing users to ask questions seamlessly in any supported language.

Multimodal Understanding

Generates descriptions for PDF images via LLaVA, supports image/chart-related queries, and ensures no loss of visual information.

Fully Offline Operation

All components run locally with no external API dependencies, zero cost, and eliminates data privacy concerns.

Tech Stack

• Document Parsing: PyMuPDF extracts text, tables, and images;
• Visual Understanding: Ollama runs LLaVA to generate image descriptions;
• Embedding Retrieval: multilingual-e5-large generates vectors, stored in Qdrant;
• Generation Layer: Ollama runs Gemma3 to generate answers;
• Interaction Layer: Gradio builds the web interface.

Processing Flow

1. Content Extraction (01_extract.py);
2. Image Captioning (02_caption.py);
3. Embedding Storage (03_embed_store.py);
4. Query Response (04_query.py).

The project supports queries in multiple languages, examples include:

• English: "What are the key findings?"
• Hindi: "इस पेपर में कौन सा एल्गोरिदम है?"
• Malayalam: "ഈ പേപ്പറിലെ മുഖ്യ കണ്ടെത്തലുകൾ എന്ത്?"
• Tamil: "முக்கிய முடிவுகள் என்ன?"
• Hinglish: "Is paper mein load balancing kaise kaam karti hai?"
• Arabic: "ما هي الخوارزمية المستخدمة؟" It covers the needs of global teams and diverse users.

The project is suitable for the following scenarios:

• Multilingual document libraries (e.g., internal documents of international organizations);
• Academic research (cross-language literature research, chart analysis);
• Enterprise knowledge bases (multilingual internal Q&A);
• Education sector (multilingual textbook understanding);
• Privacy-sensitive scenarios (local data processing in healthcare, legal fields, etc.).

The project currently has the following areas for improvement:

1. Only supports PDF format; needs to expand to other document types;
2. Complex chart understanding relies on LLaVA, and its performance needs improvement;
3. The performance stability in low-resource languages needs optimization.

The Multimodal-Multilingual-RAG project demonstrates the possibility of RAG technology evolving toward multilingual and multimodal directions. Through reasonable component selection and process design, it enables the local construction of a fully functional, language-agnostic document Q&A system, making it an excellent open-source solution for teams dealing with multilingual and multimodal documents.