RouterGym: Can Small Language Models Replace Large Language Models? A Routing-Memory Co-Design Agent Benchmark Framework

RouterGym is a benchmark framework for evaluating the feasibility of small language models (SLMs) replacing large language models (LLMs) in Agent tasks. The project implements a routing-memory co-design, supports multiple routing strategies, memory systems, and contract validation, and provides empirical evidence for SLM-led Agent architectures through comprehensive cost, quality, and latency trade-off analysis.

Large language models (LLMs) like GPT-4 and Claude are powerful but costly and slow to respond; small language models (SLMs) like Phi-3 and Mistral are low-cost, fast-responding, and easy to deploy locally. The industry has formed a new architectural pattern: most queries are routed to SLMs, and upgraded to LLMs when necessary. Based on NVIDIA Research papers, RouterGym's core question is whether SLM-led Agent architectures can match or even surpass LLM-first architectures in terms of cost, speed, factual accuracy, etc.

Intelligent Routing System

Supports three strategies: LLM-first, SLM-led, and mixture of experts, with decisions based on signals like task classification confidence and contract failure.

Memory System Layers

Includes four progressive layers: no memory, static memory, dynamic memory, and saliency-gated RAG, co-designed with routing strategies.

Contract Validation Mechanism

Ensures output conforms to expected structure through JSON Schema validation, type coercion, retry fallback, etc. Contract failure can trigger model upgrade.

Code Structure

Modular design, including directories like agents, routing, memory, contracts.

Model Configuration

Supports combinations of any 2 SLMs (e.g., Phi-3, Mistral) and 2 LLMs (e.g., GPT-4, Claude).

Evaluation Metrics

Covers multi-dimensional metrics such as factual accuracy (Groundedness), structural compliance (Schema validity), performance (Latency), and economy (Cost).

Grid search is performed on routing strategies, memory systems, model combinations, etc., using the run_grid.py tool. A typical configuration includes 3 routing strategies ×4 memory systems × contract on/off ×3 seeds, totaling 216-432 independent runs. Generated results, costs, and other data are recorded to ensure reproducibility.

When handling customer support tickets: simple queries (password reset) are directly processed by SLM; medium-complexity (function consultation) uses SLM + knowledge base retrieval; complex issues (troubleshooting) are upgraded to LLM; sensitive scenarios (security incidents) are forced to use LLM, balancing quality and cost.

Significance

Quantifies the cost-performance trade-off between SLMs and LLMs, discovers optimal routing-memory combinations, and verifies the reliability boundary of SLMs in business scenarios.

Future Directions

Support more model providers and open-source models, expand memory systems (long context, multi-modal), introduce online learning to optimize routing, and establish community benchmark datasets.

RouterGym is an important milestone in the evolution of AI Agent architectures, providing a verifiable answer to the question "Can small models handle big tasks?" As SLM capabilities improve and costs decrease, hybrid architectures led by SLMs with LLMs as a safety net may become the mainstream model for future Agent systems.