Sovereign RAG: A Hybrid Model Routing RAG System for PDF Document Analysis

Sovereign RAG is a high-performance structured Retrieval-Augmented Generation (RAG) system for PDF document analysis. Its core features include: employing an intelligent model routing strategy that calls different local models for text, tables, and scanned documents respectively; supporting fully offline operation with no data privacy risks; requiring only 4GB GPU memory for deployment, lowering hardware barriers; and enabling document question-answering capabilities suitable for various scenarios.

Currently, LLM applications are widespread, but existing RAG solutions have two major issues: first, reliance on cloud APIs leads to data privacy risks; second, they require expensive hardware resources and are difficult to deploy locally. Sovereign RAG addresses these pain points and is positioned as a fully localized, low-resource PDF document analysis system.

The system uses a hybrid model routing strategy, selecting the optimal local model for different content types:

• Text queries: Routed to Microsoft Phi-3 model, balancing reasoning ability and memory efficiency;
• Table reasoning: Routed to Alibaba Qwen2.5-3B model, which excels at structured data processing;
• Visual/scanned documents: Routed to LLaVA-Phi3 multimodal model, which can understand image content.

Tech Stack:

• Frontend: Streamlit, for quickly building interactive web interfaces;
• Vector database: LanceDB, providing efficient vector retrieval;
• Model management: Ollama, simplifying the download and invocation of local models. Resource Optimization Strategies:
• Model quantization: Reducing memory usage;
• On-demand loading: Dynamically loading required models;
• Efficient retrieval: Using LanceDB indexing;
• Document preprocessing: Parsing PDFs to separate text, tables, and images.

Privacy Protection: All data processing (parsing, embedding, reasoning) is done locally, with no reliance on external cloud services, ensuring data sovereignty. Application Scenarios:

• Enterprise internal knowledge bases: Processing policies, financial reports, etc.;
• Academic research: Retrieving paper literature;
• Personal document management: Local intelligent search;
• Offline environments: Scenarios with limited network access or high security requirements.

The project has the following limitations:

1. Local small models perform worse than cloud-based large models in complex reasoning tasks;
2. Non-English document processing capabilities need improvement;
3. The parsing accuracy of complex PDF layouts (nested tables, handwritten annotations, etc.) needs optimization.

Sovereign RAG is an attempt to develop RAG technology towards localization and privacy-first directions. Through hybrid routing and resource optimization, it achieves multimodal document question-answering under limited hardware. It is a worthy option for users who care about privacy and need local deployment. With future model iterations, it is expected to become a more mature document intelligence solution.