Private RAG Platform: A Private RAG Solution for Enterprise Sensitive Data

Private RAG Platform is an enterprise-level Retrieval-Augmented Generation (RAG) system designed specifically for handling sensitive internal documents. It supports deployment in local development environments and AWS private clouds, ensuring data never leaves the controlled environment throughout the process.

Retrieval-Augmented Generation (RAG) technology has revolutionized enterprise knowledge management, enabling non-technical staff to access an organization's accumulated internal knowledge via natural language queries. However, a core contradiction has always plagued enterprise users: how to leverage the capabilities of large language models while ensuring sensitive business data does not leak to third-party AI service providers?

Uploading internal documents to commercial APIs like OpenAI or Anthropic means data leaves the enterprise's control boundary, which is unacceptable in regulated industries such as finance, healthcare, and law. The Private RAG Platform project was born precisely to solve this fundamental problem.

This project aims to build a complete enterprise-level RAG system, with core design concepts summarized as "Data First, Local First, Private First":

1. Data First Over Model: Retrieved content is treated as data rather than instructions; the system strictly restricts LLMs to generate answers only based on retrieved content
2. Answers Must Be Verifiable: Each answer must include source references, allowing users to trace back to the specific location in the original document
3. Tenant Isolation: In a multi-tenant architecture, document access is strictly isolated to prevent cross-tenant data leakage
4. Clear Information Boundaries: When retrieved content is insufficient to answer a question, the system explicitly states "insufficient information" instead of allowing the model to improvise
5. Models Do Not Determine Data Sources: Retrieval strategies are controlled by the system; LLMs are only responsible for generating answers based on the given context

The project adopts a phased development strategy, gradually evolving from a local MVP to an AWS production environment.

Current implementation is based on the following tech stack:

• FastAPI: High-performance Python asynchronous web framework providing RESTful APIs
• PostgreSQL + pgvector: Relational database with vector extension for storing document metadata and embedding vectors
• Ollama: Local large model inference service supporting private deployment of embedding models and generation models
• Local File Storage: Documents are directly stored in the file system during the development phase

Basic RAG process has been implemented:

1. Document upload and format parsing
2. Text extraction and intelligent chunking
3. Local embedding model generates vectors
4. Vector storage and index construction
5. Semantic retrieval and relevance ranking
6. Context construction and answer generation
7. Source reference and answer return

Phase 2 focuses on the document ingestion pipeline, improving capabilities such as multi-format parsing, OCR, and table extraction.

Phase 3 optimizes retrieval and embedding, implementing advanced features like hybrid retrieval (semantic + keyword), re-ranking, and query rewriting.

Phase 4 enhances answer generation, introducing interactive capabilities such as reference verification, multi-turn dialogue, and follow-up clarification.

Phase 5 completes AWS cloud-native deployment, with the target architecture including:

• Network Layer: VPC divided into public and private subnets; FastAPI services deployed in private subnets
• Storage Layer: RDS PostgreSQL managed database; S3 private buckets for storing original documents
• Computing Layer: EC2 GPU instances running Ollama or vLLM, or connecting to Amazon Bedrock private endpoints
• Security Layer: Secrets Manager for credential management; Security Groups and IAM roles for access control
• Monitoring Layer: CloudWatch for log and metric collection

Banks and securities firms can use this platform to build intelligent Q&A systems for internal compliance documents, research reports, and customer data, ensuring sensitive financial data never leaves the enterprise network throughout the process.