GenAI Document Assistant: Enterprise-Grade Multi-Agent Document Q&A System

Title: GenAI Document Assistant: Enterprise-Grade Multi-Agent Document Q&A System Original Author: rpatelvns Source Platform: GitHub Original Link: https://github.com/rpatelvns/genai-doc-assistant Publication Date: May 31, 2026

GenAI Document Assistant is an enterprise-grade generative AI document processing and knowledge extraction assistant based on the LangGraph multi-agent orchestration framework. It supports multiple formats such as PDF, Word, and Excel. Through a complete workflow of Retrieval-Augmented Generation (RAG) and automatic response verification, it effectively reduces model hallucinations and ensures the factual accuracy of answers. The project uses local embedding processing to protect data privacy, supports switching between multiple LLM providers like OpenAI, Anthropic, Google Gemini, and Groq, and is suitable for scenarios such as internal enterprise knowledge base Q&A and compliance reviews.

Project Background and Overview

GenAI Document Assistant is an enterprise-grade generative AI document processing and knowledge extraction assistant. It uses advanced AI models and multi-agent orchestration frameworks to automate complex document tasks such as information extraction, summary generation, content creation, document understanding, and context-aware Q&A.

Unlike traditional simple RAG systems, this project adopts a three-layer agent architecture and adds an independent verification link after generating answers, effectively identifying and reducing model hallucinations to ensure the factual accuracy of responses.

Core Architecture and Processing Flow

Multi-Agent Orchestration Framework

Based on LangGraph, a stateful and modular multi-agent workflow is built, where state flows sequentially between agent nodes. The system is divided into three core processing stages:

1. Retrieval Agent: Receives user queries, uses HuggingFace's nomic-embed-text-v1.5 model for local vectorization, performs similarity searches in ChromaDB, and returns relevant document fragments and source metadata.
2. Generation Agent: Constructs structured prompts based on retrieval context and original queries, calls LLMs to generate answers with reasoning processes, and supports switching between multiple LLM providers.
3. Verification Agent: Acts as a "fact checker" to evaluate whether answers are supported by the context, generates detailed verification reports, and solves the hallucination problem of RAG systems.

Document Processing Pipeline

Original Document → MarkItDown Conversion → Markdown Text → Text Chunking → Vectorization → ChromaDB Storage

The MarkItDown tool supports formats such as PDF, Word, Excel, PPTX, TXT, and images (OCR required), and uniformly converts them to Markdown for easy LLM understanding.

Technology Stack and Privacy & Security Design

Detailed Technology Stack

Component Category	Technology Selection	Reason for Selection
Frontend Interface	Streamlit	Quickly build interactive web applications, supporting chat interfaces and expandable details
Agent Orchestration	LangGraph + LangChain	Supports complex state machine workflows and modular agent design
Vector Database	ChromaDB	Lightweight, local-first, suitable for prototyping and enterprise deployment
Embedding Model	HuggingFace (nomic-embed-text-v1.5)	Runs locally, privacy-first, and has excellent performance
Document Processing	Microsoft MarkItDown	Official Microsoft tool with comprehensive format support
LLM Interface	Multi-provider support	Flexible switching to avoid vendor lock-in

Privacy and Security Design

1. Data Privacy Protection: Local embedding processing (original documents do not leave the local environment), temporary storage of API keys (no persistence), and only sending retrieval fragments to third-party APIs.
2. Security Notes: Sensitive data requires signing a DPA or zero-retention policy; local storage needs access control and encryption; the current version has no user login/RBAC.

Usage and Interaction Design

Configuration Panel (Sidebar)

• Provider Selection: Dropdown menu to select LLM providers (OpenAI/Anthropic/Google/Groq)
• Model Selection: Choose specific models (e.g., GPT-4o-mini, Claude-3.5-Sonnet)
• API Key Input: Securely enter the corresponding provider's API key
• Document Upload: Support multi-file upload, with background automatic processing and indexing

Chat Interface (Main Window)

After the user enters a question, the system executes the complete RAG + verification process. Below the answer, three expandable areas are provided:

• Reasoning and Thought Chain: Displays the model's thinking process
• Verification Summary: Shows the evaluation results of the verification agent
• Source Citations: Lists the document fragments and filenames used

Limitations and Improvement Directions

Current Limitations

1. Context window restriction: Overly large documents or too many retrieval fragments may exceed the LLM token limit
2. Stateless conversation memory: Multi-turn conversation memory is not fully utilized
3. Local vector storage scalability: ChromaDB is file-based and not suitable for large-scale enterprise deployment
4. Document parseability: Limited effectiveness for pure scanned PDFs or image-intensive documents

Potential Improvements

• Integrate OCR processing for scanned documents
• Implement multi-turn conversation memory
• Add user authentication and RBAC
• Support multi-language document processing
• Integrate more vector database options (Pinecone, Weaviate, etc.)

Application Scenarios and Project Value

GenAI Document Assistant represents a mature implementation paradigm for enterprise-grade RAG systems and is suitable for the following scenarios:

• Internal enterprise knowledge base Q&A: Employees quickly query company documents and policies
• Research report analysis: Cross-document comprehensive Q&A
• Compliance review: Query contracts and regulatory clauses
• Customer support enhancement: Find accurate answers from product documents

The core value lies in introducing a multi-agent verification mechanism into the RAG process, ensuring answer quality while providing traceable verification reports, which is crucial for enterprise-level applications.