RouterGym: A Systematic Evaluation Framework for Whether Small Language Models Can Replace Large Language Models

RouterGym is an open-source framework designed to systematically assess whether small language models (SLMs) can replace large language models (LLMs) in agentic AI tasks. It uses a routing-memory co-design approach to measure trade-offs between cost, quality, and latency. This post breaks down its background, architecture, evaluation methods, applications, and insights.

LLMs like GPT-4 excel in agent tasks but are costly and slow. SLMs (e.g., Phi-3, Mistral) are cheaper, faster, and easier to deploy privately but less capable. The core question: Can SLMs handle most work with LLM calls only when necessary? RouterGym was created to answer this by quantifying if SLM-dominant agent architectures can match or outperform LLM-first ones. It is part of Kparobor Akpomiemie's degree thesis and aligns with NVIDIA's view that SLMs are the future of agents.

RouterGym's architecture decouples agent components into configurable modules:

1. Routing: Decides SLM/LLM use. Strategies: LLM-first (safe but expensive), SLM-dominant (fallback to LLM on low confidence/contract failure/safety risks), Hybrid specialist (domain-specific SLMs + LLM as fallback).
2. Memory: Manages context injection. Strategies: None (no extra context), Static (fixed system prompts), Dynamic (RAG), Salience-gated RAG (relevant context only).
3. Contracts: Ensures output quality via JSON Schema validation and structured retries (fail → upgrade to stronger model).

RouterGym evaluates beyond accuracy, covering:

• Performance: Groundedness (factuality), Schema Validity (format compliance), Task Accuracy.
• Cost/Efficiency: Latency, Cost (token-based), Fallback Rate (SLM→LLM). It uses run_grid.py for systematic experiments (combining routers, memories, SLMs/LLMs) and analyzer to generate reports.

RouterGym's customer service ticket example shows routing in action:

• Simple queries (e.g., password reset) → SLM (Phi-3) if confidence high (0.92), validated via contracts.
• Complex tech issues (e.g., DB connection timeouts) → LLM (GPT-4) if confidence low (0.67). This balances quality and cost.

Key findings:

1. Cost-quality trade-offs are quantifiable (Pareto optimal configs exist).
2. Contracts are critical for SLM reliability (reduces instability to measurable fallback rates).
3. Memory and routing must be co-designed (e.g., SLM-dominant needs salience-gated RAG). Industry impact: Enables data-driven architecture decisions, promotes "right model for right task" thinking, and supports edge/private deployments.

Current limitations:

• Limited model provider support (mostly OpenAI/Anthropic; less open-source local deployment).
• Narrow task coverage (focus on customer service, less on code/creative writing).
• Coarse latency measurement (no component-wise breakdown). Future directions: Multi-modal support, online learning for dynamic routing, federated evaluation (privacy-preserving), better visualization tools.