DocQuery: A Localized RAG Document Query System Based on NVIDIA DGX Spark

DocQuery is a localized RAG document query system based on NVIDIA DGX Spark, built with C#/.NET 8 and React. It aims to address data privacy issues, supports local large language model inference, and provides users with a secure and efficient intelligent document query experience. The system fully leverages the edge AI computing capabilities of DGX Spark to enable fully offline production-grade model operation.

DocQuery was born out of considerations for data sovereignty and privacy protection. Traditional cloud-based document Q&A systems struggle to meet compliance requirements in industries like finance and healthcare. For technology selection: the backend uses C#/.NET 8 (balancing performance, enterprise ecosystem, cross-platform support, etc.), the frontend adopts React (smooth interaction and scalability), and the core inference platform is NVIDIA DGX Spark (a desktop AI supercomputer that supports offline operation of large models).

DocQuery follows the classic RAG paradigm with optimizations:

1. Document Ingestion: Supports formats like PDF/Word, including PDF parsing, optional OCR, cleaning, and chunking.
2. Vectorization Indexing: Converts text to vectors using open-source embedding models, stores them in a local vector database, and uses DGX Spark's parallel computing for fast processing.
3. Retrieval and Sorting: Combines semantic retrieval, keyword matching, and hybrid sorting, then re-ranks to filter relevant segments.
4. Answer Generation: Calls local open-source models (Llama/Mistral/Qwen, etc.), supports streaming output, and DGX Spark ensures low latency.

DGX Spark integration involves:

1. Model Optimization: INT8/INT4 quantization reduces memory usage and supports model sharding.
2. Inference Acceleration: Uses TensorRT-LLM to improve throughput and dynamic batching to enhance hardware utilization.
3. Resource Management: Fine-grained monitoring of memory/vRAM, dynamic resource allocation, and automatic unloading of model parameters to free up vRAM when the load is low.

Application scenarios include enterprise knowledge management (internal document query), personal knowledge base (cross-document association), and compliance industries (healthcare/legal/finance). Deployment modes: standalone deployment (for individual users), server-client architecture (multi-user access in enterprise LAN).

DocQuery uses a modular design, compatible with the Hugging Face Transformers ecosystem (seamless integration of new models), supports pluggable vector databases (built-in lightweight options and professional libraries like Milvus/Weaviate), has componentized and customizable front-end for extension, and welcomes community contributions.

DocQuery represents the evolution direction of localized AI. The improvement of edge hardware and the development of open-source models make it production-ready. This trend reduces privacy/cost concerns and gives users control. Future directions include multi-modal integration, efficient model compression, and user-friendly deployment toolchains. Localized AI will complement cloud services to build a diverse and resilient ecosystem.