Implementation of an Intelligent Document Q&A System Based on RAG and Local LLM

This article introduces a complete RAG (Retrieval-Augmented Generation) document Q&A system that uses the FAISS vector database, Sentence Transformers embedding model, and Ollama local large language model to implement intelligent Q&A functionality for PDF documents.

• Original Author: Ankita Dnyanoba Shinde
• Source Platform: GitHub
• Original Title: Question_Answering_System_Using_RAG_LLM
• Original Link: https://github.com/theankita/Question_Answering_System_Using_RAG_LLM
• Publication Date: May 27, 2026

In today's era of rapid AI development, how to enable large language models to accurately answer questions based on specific document content without generating "hallucinations" is an important technical challenge. The open-source project introduced in this article provides a complete solution: an intelligent document Q&A system based on the Retrieval-Augmented Generation (RAG) architecture, which runs entirely locally without relying on cloud APIs.

This system allows users to upload PDF documents and then ask questions based on the document content. The system retrieves the most relevant paragraphs from the document and uses a locally running large language model to generate accurate answers. The entire process combines the precision of vector retrieval with the flexibility of generative AI.

The core idea of the RAG architecture is to combine information retrieval with text generation. While traditional language models have extensive knowledge, they tend to produce inaccurate information. RAG significantly improves the accuracy and traceability of answers by first retrieving relevant context from the knowledge base and then allowing the model to generate answers based on that context.

The RAG workflow of this project is as follows:

1. PDF Document Upload - Users upload PDF files via the Streamlit interface
2. Text Extraction - Use the PyPDF2 library to extract readable text from PDFs
3. Intelligent Chunking - Split long text into 500-character paragraphs with 100-character overlaps to ensure semantic coherence
4. Embedding Generation - Convert text into dense vector representations using the all-MiniLM-L6-v2 model
5. Vector Storage - Store embedding vectors in a FAISS index to support fast similarity search
6. Question Embedding - Convert user queries into the same vector space
7. Semantic Retrieval - Find the most relevant document fragments in FAISS
8. Context Combination - Combine retrieved fragments into a context prompt
9. Answer Generation - Call the Llama3 model via Ollama to generate the final answer

Vector Database: FAISS

FAISS (Facebook AI Similarity Search) is an efficient similarity search library developed by Meta. It can quickly find the vectors most similar to the query among massive vectors, making it an ideal choice for building RAG systems. This project uses the CPU version of FAISS and can run without a GPU.

Embedding Model: Sentence Transformers

The project uses the all-MiniLM-L6-v2 model to generate text embeddings. This is a lightweight yet effective sentence embedding model that maps semantically similar text to adjacent vector spaces. The model is only about 80MB in size, making it perfect for local deployment.

Local LLM: Ollama + Llama3

Ollama is a tool that simplifies running local large language models. This project uses the Llama3 model, which performs inference entirely locally without network connectivity, protecting data privacy. Through carefully designed prompts, the model is ensured to answer questions only based on the provided context, avoiding hallucinations.

Interactive Interface: Streamlit

Streamlit is a Python library for quickly building data applications. This project uses it to create a clean, modern web interface with features including PDF upload, text preview, chunk statistics, context viewing, and real-time Q&A.

Unlike solutions that rely on cloud services such as OpenAI API or Claude, all components of this system run locally. This means:

• Data Privacy: Sensitive documents never leave the local machine
• Zero API Cost: No need to pay pay-as-you-go fees
• Offline Availability: Usable without an internet connection
• Customizability: Can be replaced with any Ollama-compatible model

The system is not just a simple full-text search. Through semantic embedding and vector retrieval, it can understand the deep meaning of queries and find content that is conceptually related but uses different wording. The overlapping chunking strategy ensures that context across paragraph boundaries is not missed.

The project has designed a dedicated prompt template that requires the model to:

• Only use the provided context information
• Avoid generating guesses outside the context
• Produce structured and clear answers

This design significantly reduces the probability of large language models "speaking nonsense with a straight face".