Building an Enterprise-Grade RAG System: In-Depth Practice of Hybrid Retrieval and Re-Ranking

This article provides an in-depth analysis of a production-grade RAG system implementation based on the MS MARCO dataset, covering the complete tech stack of dense retrieval, BM25 sparse retrieval, cross-encoder re-ranking, as well as engineering practices for FAISS index optimization and latency tracking. It addresses key challenges of RAG systems such as knowledge timeliness, hallucination issues, and domain adaptation difficulties.

RAG has become a standard component in enterprise AI, combining external knowledge bases to solve issues like LLM knowledge timeliness and hallucinations. However, building a production-ready RAG system faces three core challenges: balancing retrieval quality and efficiency (pure vector and BM25 each have their pros and cons), precision of result ranking, and constraints on system latency and throughput.

We adopt complementary dense retrieval (pre-trained model encoding vectors, FAISS index supporting fast approximate search) and BM25 sparse retrieval (keyword exact matching); fusion mechanisms include linear weighting, RRF, etc., to ensure independent optimization and a concise architecture.

After recalling candidates in the retrieval phase, cross-encoders (single-tower architecture for deep interaction between queries and documents) are used for precise sorting; the two-stage architecture (fast recall + fine sorting) balances accuracy and efficiency, and cross-encoders have higher accuracy than dual-tower models.

Core evaluation metrics include Recall@K (retrieval completeness), MRR (position of the first relevant document), and NDCG (ranking quality); the MS MARCO dataset (real search scenarios with manual annotations) is selected to verify the system's effectiveness.

End-to-end latency tracking is implemented, covering links such as index loading, retrieval, and re-ranking; optimization methods include FAISS index quantization, batch inference to improve GPU utilization, and caching of popular query results.

RAG optimization requires multi-dimensional collaboration (retrieval strategy, ranking model, evaluation, engineering); future directions include multi-modal retrieval, adaptive strategies, end-to-end optimization, etc., and the classic architecture serves as the foundation for advancement.