LLM Retrieval Strategy Benchmark Framework: A Comprehensive Comparison of Azure AI Search and GraphRAG Performance

This article introduces an open-source LLM retrieval strategy evaluation framework llm-retrieval-benchmark, which supports multiple retrieval modes including Azure AI Search hybrid search, semantic ranking, and GraphRAG. Through standardized evaluation, it helps developers select the optimal retrieval solution based on query types, addressing the problem of relying on subjective experience for RAG technology selection.

With the popularization of LLM applications, Retrieval-Augmented Generation (RAG) has become a core technology to solve model hallucinations and knowledge timeliness issues. However, when faced with multiple backend options such as traditional vector search and emerging graph retrieval, development teams often lack systematic comparative evaluations and rely on subjective experience rather than objective data for technology selection, making it difficult to balance performance, accuracy, and cost.

llm-retrieval-benchmark is an open-source benchmark framework maintained by developer xenakal, designed specifically for evaluating LLM retrieval strategies. Its core value lies in query classification awareness—it not only reports overall metrics but also breaks down performance by query type, helping developers understand the performance of different strategies in scenarios such as factual questions and complex problems.

The framework covers mainstream RAG technical routes:

1. Azure AI Search: A hybrid mode combining BM25 keyword matching and vector semantic search, with semantic reordering to refine results, excelling at handling queries with technical terms or ambiguous words;
2. GraphRAG: An open-source knowledge graph RAG from Microsoft Research, supporting three modes: global (macro theme), local (entity neighbors), and drift (multi-hop reasoning), suitable for reasoning and correlation analysis scenarios;
3. Custom Backend: Reserved extension interface allowing integration of self-owned retrieval implementations to meet customized needs.

The framework uses multi-dimensional quantitative metrics:

• Precision and Recall: Reports P-R curves at different thresholds to help balance the proportion of relevant results and coverage;
• Mean Reciprocal Rank (MRR): Focuses on the position of the first relevant result, which is crucial for question-answering systems;
• Query Category Breakdown: Reports metrics by query categories including factual (Who/What/When), explanatory (Why/How), comparative, and aggregative queries, identifying the comfort zones and blind spots of each strategy.

The framework provides RAG teams with:

• Technical Selection Basis: First-hand data from self-owned datasets to avoid blind following;
• Performance Regression Detection: Establishes baselines to ensure that iterations do not reduce retrieval quality;
• Cost-Benefit Analysis: Combines quantitative metrics with cost data to support ROI decisions;
• Academic Research Tool: Standardized evaluation environment to improve the comparability and reproducibility of research results.

When using the framework, note the following:

• Dataset Dependency: Evaluation results are affected by the quality and representativeness of the test query set; if there is a large difference from the production distribution, predictions will be inaccurate;
• Annotation Cost: High-quality relevance annotations require manual input, making large-scale evaluation costly;
• Dynamic Environment: Retrieval service performance changes with index updates and model iterations; a single evaluation cannot reflect long-term performance.

llm-retrieval-benchmark provides an objective quantitative tool for RAG technology selection, which is of significant value in today's rapidly evolving retrieval technology landscape. In the future, we look forward to expanding its capabilities to evaluate multi-modal, real-time, and personalized retrieval. It is recommended that relevant teams fork the project and test it on their own data to gain targeted insights.