SLM-LLM Intelligent Routing System: How to Achieve 13x Performance Improvement with Confidence Gating

This article introduces the SLM-LLM intelligent routing system developed by Venisa at Manipal Institute of Technology. It dynamically routes queries to SLMs or LLMs via a confidence gating mechanism, resolving the contradiction enterprises face between high cost and slow response of large models and insufficient capabilities of small models. This achieves triple optimization of cost, latency, and performance—with up to 13x acceleration in specific scenarios.

With the widespread application of LLMs (e.g., GPT-4, Mistral7B), enterprises face a core contradiction: LLMs have strong capabilities but are high-cost and slow to respond; SLMs are cheap and fast but perform poorly on complex reasoning tasks. The traditional "one-size-fits-all" use of LLMs leads to resource waste. The core problem the routing system needs to solve is how to route simple queries to SLMs and complex ones to LLMs without sacrificing quality.

The system uses a three-stage processing pipeline:

1. Symbolic Math Engine: Handles mathematical expressions with ~1ms response time;
2. NanoQA Small Model (135M parameters): Processes factual short-answer queries, trained on 300k+ QA pairs using Focal Loss (γ=2) and GPT-2 knowledge distillation;
3. Mistral7B Large Model: Serves as a fallback for complex reasoning tasks. Routing decisions rely on confidence gating: Calculate the average softmax probability of generated tokens—if ≥0.6, use SLM output; if <0.6, escalate to LLM. No additional classifiers or labeled data are needed.

Training aspects:

• Dataset: Built 300k+ QA pairs (manually curated, augmented training, domain-specific data);
• Techniques: Used Focal Loss with γ=2 to address class imbalance, distilled knowledge from GPT-2 to NanoQA, and performed token-level fine training to enhance semantic sensitivity.

The system has excellent performance metrics:

Metric	Value
Accuracy	98.0%
MRR	98.6%
Routing F1 Score	82.1%
Total Response Time Reduction	63%
Acceleration vs. Pure LLM Solution	~13x
Data shows that while maintaining high-quality output, the system significantly reduces latency and cost, with high accuracy in routing decisions.

Application value is reflected in three aspects:

1. Cost Optimization: SLM call cost is only 1/10 of LLM; most simple queries routed to SLMs significantly reduce expenses;
2. Latency Improvement: 63% reduction in response time enhances user experience (e.g., real-time dialogue, customer service robots);
3. Local Deployment: Supports Ollama integration with Mistral7B for local operation, meeting privacy compliance requirements of data-sensitive industries like finance and healthcare.

Current system limitations and improvement plans:

• Need to enhance understanding of synonyms and paraphrasing (plan to introduce embedding technology);
• NanoQA can be scaled to larger parameter sizes;
• Integrate reinforcement learning to optimize routing strategies.