SLM-to-LLM Routing System: Finding the Optimal Balance Between Cost and Performance

This article introduces the SLM-to-LLM intelligent routing system, which can automatically schedule between Small Language Models (SLMs) and Large Language Models (LLMs) based on query complexity. It significantly reduces inference costs while ensuring response quality, making it a key optimization strategy for enterprises to control costs and enhance user experience when deploying AI at scale.

With the widespread application of LLMs, enterprises need to balance output quality and inference costs: LLMs (such as GPT-4, Claude3) have excellent performance but are high-cost and high-latency; SLMs (such as Phi-3, Gemma2B) are fast and low-cost when handling simple tasks. This discrepancy has spurred the demand for an intelligent routing system that can automatically determine which model to use for a query.

System Architecture

The SLM-to-LLM router is a classification decision system with the following workflow: 1. Receive the request; 2. Evaluate query complexity, domain professionalism, and task type; 3. Select the appropriate model; 4. Return the response.

Core Routing Dimensions

• Query complexity: Use SLMs for simple tasks (FAQs, basic translation), and LLMs for complex tasks (multi-step reasoning, code generation);
• Cost sensitivity: Prioritize SLMs for batch scenarios, and LLMs for critical decision-making scenarios;
• Latency requirements: Use SLMs for real-time interactions, and LLMs are acceptable for offline analysis (which tolerates higher latency).

Rule-Based Routing

Implemented via keyword matching (e.g., routing queries containing "code" or "algorithm" to LLMs), it is simple but has limited flexibility.

Semantic Routing Based on Embedding Vectors

Uses embedding vectors to calculate semantic similarity and matches historical complexity data to predict the complexity level.

Machine Learning Classifiers

Train lightweight models (such as BERT, logistic regression) to predict model selection, enabling continuous learning and optimization.

Take a scenario handling 100,000 queries per day as an example:

• Using LLMs exclusively: $0.02 per query, daily cost of $2000;
• Using the routing system: 70% of simple queries use SLMs ($0.001 per query), 30% use LLMs;
• Optimized cost: 70000×0.001 + 30000×0.02 = $670, saving approximately 66%.

Misjudgment Issues

• Set a confidence threshold; route to LLMs by default for low-confidence cases;
• Establish a user feedback mechanism to adjust strategies;
• Automatically monitor the quality of SLM outputs.

Model Management Complexity

Adopt a Model-as-a-Service (MaaS) architecture to manage multiple models via a unified interface.

Latency Trade-off

Cache routing decisions for common queries to reduce routing overhead.

Future routing systems may:

• Support more model tiers (Tiny, Small, Medium, Large);
• Introduce dynamic model combinations where multiple SLMs collaborate to complete complex tasks;
• Implement personalized routing, optimizing selection based on users' historical preferences.

The SLM-to-LLM routing system does not pursue the extreme performance of a single model; instead, it maximizes overall efficiency through intelligent orchestration. It is a key strategy for enterprises to control costs and enhance user experience when deploying LLMs at scale.