OmniBench-RAG: A Multi-Domain Comprehensive RAG Evaluation Platform for Large Language Models

OmniBench-RAG is a comprehensive Retrieval-Augmented Generation (RAG) evaluation platform designed specifically for Large Language Models (LLMs). Unlike static benchmarks, it features dynamic dataset generation, the ability to evaluate across 9 professional domains, focuses on accuracy and efficiency metrics, provides custom document upload and visual analysis functions, and offers a flexible and reproducible testing environment for researchers and developers.

Most existing LLM evaluation benchmarks rely on fixed datasets, which carry the risk of data leakage and are difficult to adapt to new evaluation needs. OmniBench-RAG aims to address this issue by using dynamic dataset generation to mitigate evaluation bias and meet the needs of cross-domain, multi-dimensional RAG scenario evaluation.

OmniBench-RAG supports evaluation in 9 professional domains including geography, history, and health, with each domain having its own knowledge graph built based on Wikidata. Its core innovation lies in dynamic dataset generation: it automatically extracts entity relationships from Wikidata, generates domain-specific reasoning rules, and constructs dynamic evaluation datasets, effectively avoiding data leakage.

The platform provides a complete RAG testing workflow: it supports custom PDF document upload, intelligent text chunking, FAISS vector index construction, and configuration of multiple retrieval parameters. It also has a 'strong RAG material' comparison function to quantify the value of the RAG mechanism. The system uses a modular architecture, including Flask backend services, a data processing layer (PDF extraction, FAISS indexing, etc.), a Prolog reasoning engine, and a frontend interface.

Evaluation metrics include: 1. Accuracy evaluation: Using a fine-tuned model to perform binary classification on answer correctness, supporting multiple question types such as reverse reasoning and negative reasoning; 2. Efficiency tracking: Real-time monitoring of memory usage, response time, and GPU utilization; 3. Visual analysis: Automatically generating multi-domain radar charts to show performance differences, and providing statistical aggregation analysis such as average accuracy and improvement rate.

The platform is suitable for: Model selection (cross-domain multi-metric comparison), RAG process optimization (testing the impact of retrieval strategies, etc.), academic research (reproducible evaluation environment), and domain adaptation evaluation (custom vertical domain document upload).

The platform supports flexible deployment (from local to production) and intelligently adapts to CUDA GPUs, Apple MPS, or CPUs. It provides a quick start guide and API documentation for easy integration. OmniBench-RAG fills the gap in comprehensive evaluation tools for RAG scenarios, and its importance will become increasingly prominent as RAG technology becomes more widespread.