Coverage Illusion: Query Enhancement Cost Optimization and Post-Retrieval Cascade Strategy in Production-Grade RAG Systems

This article takes the production-grade RAG system of the Danish National Encyclopedia as a case study to reveal the Coverage Illusion phenomenon—synthetic queries overestimate the need for LLM enhancement. The proposed post-retrieval cascade strategy achieves a 31.8% latency reduction, 72.2% of queries not requiring LLM enhancement, and improves system quality with zero training cost.

Modern RAG systems commonly use query enhancement techniques like HyDE to improve retrieval coverage, but there are two major issues:

1. Each enhancement call to LLM leads to staggering costs at scale;
2. LLM calls increase end-to-end latency, affecting user experience. More importantly, the "one-size-fits-all" enhancement strategy lacks empirical basis—does every query need expensive enhancement?

The research team analyzed over 20,000 query-workflow pairs from the Danish National Encyclopedia and found:

• Synthetic query tests show that 90% require LLM enhancement;
• Only 27.8% of queries in real production traffic actually need enhancement. This gap reveals the mismatch between synthetic data and real user behavior—synthetic queries are more complex and ambiguous, while real queries are more direct and clear.

We attempted to build pre-retrieval routers using four machine learning paradigms such as classifiers and regression models, but the results show that it is impossible to reliably predict whether enhancement is needed based solely on query text. The reason is that the "enhancement need" of a query is a function of the index content—the same query may have different needs in different indexes, which must be determined after retrieval.

The core mechanism follows the "cheapest first" principle:

1. First layer: Direct retrieval (no enhancement, lowest cost and latency);
2. Second layer: Trigger HyDE enhanced retrieval only when the first layer returns empty documents;
3. Optional extension: Add stronger enhancement methods such as query expansion. Advantages: Zero training cost, no need for auxiliary infrastructure, simple implementation and low deployment cost.

Results in the Danish production environment:

Metric	Post-Retrieval Cascade	Always-HyDE	Improvement
Comprehensive Quality Score	+0.140	Baseline	+0.140
End-to-End Latency	-31.8%	Baseline	31.8% reduction
Proportion of Queries Without LLM Enhancement	72.2%	0%	Significant increase
Reason for quality improvement: Avoid noise introduced by unnecessary enhancements and reduce deviations from user intent.

1. Beware of Synthetic Data Misleading: Systems designed based on synthetic queries may perform very differently in real environments, so production traffic should be used for evaluation;
2. Delayed Decision-Making is Better Than Premature Optimization: Delay decision-making until sufficient information is available (after retrieval), similar to "lazy evaluation" in software engineering;
3. Simple Strategies Outperform Complex Models: The zero-training cascade strategy is better than multiple machine learning routing schemes;
4. New Cost-Quality Trade-off: Intelligent resource allocation can reduce costs and improve quality at the same time.

Coverage Illusion reveals that the over-reliance of RAG systems on query enhancement stems from a misunderstanding of real user behavior. The post-retrieval cascade strategy provides a zero-training, easy-to-implement solution that improves efficiency and experience. Limitations: Dependent on basic retrieval quality; simple retrieval has low hit rates in some fields (e.g., professional technical documents); cascade depth thresholds need scenario-specific tuning. Future directions: Explore fine-grained cascade strategies (dynamic decision-making based on retrieval result quality) and extend to RAG components such as re-ranking and context compression.