Learning Agent Routing from Early Experience: The Efficient Query Allocation Strategy of BoundaryRouter

This paper proposes the BoundaryRouter framework, which addresses the problem of efficient routing between LLM inference and full agent execution in cold-start scenarios by leveraging early behavioral experience and scoring-criteria-guided reasoning. Experiments show that this strategy can reduce inference time by 60.6% while improving performance by 28.6%.

Large language model agents excel at complex reasoning tasks but have high latency and computational costs; many queries do not require full agent execution, and the mismatch in resource requirements has spurred the need for intelligent routing. Cold-start scenarios lack historical data, making traditional supervised learning methods difficult to apply, requiring routing solutions that are training-free, quickly deployable, and capable of gradual optimization.

The core innovation of BoundaryRouter is 'learning from early experience': it builds a compact experience memory bank, obtains initial comparative experience by simultaneously executing lightweight LLM and full agent through a shared seed set; during the inference phase, it retrieves similar historical cases to guide routing decisions, combined with scoring-criteria-guided reasoning, which is transparent and interpretable, making it suitable for cold-start scenarios.

The research team constructed the RouteBench benchmark, covering three key scenarios: in-domain, paraphrased, and out-of-domain, ensuring that evaluation results reflect the method's robustness under different distribution shifts and providing a standardized evaluation platform for agent routing research.

Compared to the baseline of always using agents, BoundaryRouter reduces inference time by 60.6% and improves performance by 28.6% compared to direct LLM inference; it improves by 37.9% compared to prompt-based routing and by 8.2% compared to retrieval-only routing, demonstrating the advantages of combining experience learning and scoring guidance.

The BoundaryRouter methodology has broad reference significance: the paradigm of small-data startup and continuous optimization is applicable to multiple AI deployment scenarios; it provides a feasible solution for developers to launch quickly without large amounts of historical data and improve gradually with use, which meets real-world deployment needs.

Current limitations: The capacity and retrieval efficiency of the experience memory bank may become bottlenecks in large-scale scenarios, and the similarity measurement is not optimal. Future directions: Explore efficient memory structures, accurate similarity calculation, dynamically update the memory bank to adapt to distribution shifts, and expand to multi-agent scenarios.