Can QPP Select the Optimal Query Variant? Uncovering Systematic Discrepancies Between Retrieval and Generation Goals in RAG Pipelines

This study focuses on the challenge of query variant selection in RAG systems: generating multiple variants can improve recall, but the computational cost is high. The research introduces Query Performance Prediction (QPP) technology to explore its value in intra-topic variant selection. Key findings: Variants that maximize retrieval metrics (e.g., nDCG) do not necessarily produce the best answers, indicating a "utility gap"; however, lightweight pre-retrieval predictors can effectively improve end-to-end RAG quality.

In RAG systems, query variants generated by LLMs can retrieve information from multiple perspectives, but the cost of full execution is extremely high. Traditional QPP is used for cross-topic query difficulty estimation; this study raises a new question: Can QPP be used for variant selection within the same information need? Unlike cross-topic discrimination, fine-grained discrimination within a topic (different expressions of the same need) has more practical value for RAG optimization.

The experiment is based on the TREC-RAG benchmark (real scenarios, multi-document retrieval, end-to-end generation evaluation). Multiple semantically equivalent query variants are generated (e.g., different expressions of the original query "Impact of climate change on agriculture"). QPP predictors are divided into two categories: pre-retrieval (based on query features, low overhead) and post-retrieval (based on retrieval results, high cost). Evaluation uses relevance metrics (Pearson/Spearman coefficients) and decision metrics (selection accuracy, performance improvement).

Key finding: There is a systematic discrepancy between retrieval metrics and generation quality—variants that maximize nDCG often fail to generate the best answers, which is the "utility gap". Reasons include: 1. Relevance ≠ information value (highly relevant documents may be redundant or lack key details); 2. Generation requires diversity (redundant documents are of limited help for generation); 3. Sensitivity to ranking position (different orderings affect generation quality).

Although QPP cannot select the absolute best variant, it often outperforms the original query. Unexpected finding: Lightweight pre-retrieval predictors can match or even surpass expensive post-retrieval methods due to their low latency, low cost, and scalability. Both sparse and dense retrievers have a utility gap; the gap is larger for dense retrievers, but QPP is effective for both types.

1. Jointly optimize retrieval and generation (using generation-aware metrics, end-to-end architecture); 2. Use QPP as a cost-effective tool (select variants when resources are limited); 3. Optimize variant generation (guided by QPP, diversity, adaptive quantity); 4. Evolve evaluation metrics (unify metrics for retrieval and generation).

Limitations: Scenario limitations of the TREC-RAG dataset, single variant generation method, traditional QPP framework, static evaluation. Future directions: Develop generation-aware QPP, end-to-end variant selection models, multi-turn RAG variant strategies, and larger-scale benchmarks.