Semantic LLM Router: An Intelligent Inference Routing System Based on Auction Mechanism

This article introduces Semantic LLM Router, a semantic routing system supporting self-hosted LLM inference clusters. The system innovatively incorporates auction mechanisms to achieve multi-dimensional optimization of cost, latency, accuracy, and energy consumption. It supports mainstream inference frameworks such as vLLM, NVIDIA Dynamo, and Ray Serve, and features user preference management, self-correcting latency reputation system, accuracy sampling monitoring, etc., providing a solution to the resource scheduling challenges of self-hosted LLM clusters.

With the widespread application of LLMs in enterprises, resource scheduling for self-hosted inference clusters has become a core operational challenge. Traditional load balancing solutions (round-robin, least connections) cannot handle the complex trade-offs between cost, latency, and accuracy in LLM inference. The semantic-llm-router project developed by yfan000 provides an innovative solution to this problem through auction mechanisms.

The core of the system is an auction-based bidding mechanism, where each model instance actively participates in bidding and quotes on four dimensions based on real-time status:

• Cost: Estimate resource consumption cost based on KV cache hit rate and computational load;
• Latency: Provide response time commitments based on request queue depth and estimated token count;
• Accuracy: Quantify competence through historical performance and task matching degree;
• Energy Consumption: Consider the energy consumption of requests to support green computing needs.

The system adopts a dynamic pricing strategy, using KV cache hit rate and request queue length as load signals: when the cache hit rate is high, it lowers the bid to attract similar requests; when the queue is backlogged, it raises the bid to guide traffic to other instances, achieving cluster-level load balancing and avoiding hot spot issues.

The system supports three preset user modes:

• Accuracy Priority: Prioritize high-performance models, suitable for scenarios such as code generation and document writing;
• Economic Mode: Select the instance with the highest cost-performance ratio, suitable for batch processing and non-critical tasks;
• Eco-Friendly Mode: Prioritize low-energy paths to meet sustainable development needs. In addition, it supports fine-grained budget management, allowing users to configure upper limits for token and energy consumption budgets to prevent resource abuse.

Latency Reputation System: Track model latency performance based on Exponential Moving Average (EMA), record prediction deviations and adjust bid weights, reducing the priority of models that frequently overcommit in latency-sensitive requests. Accuracy Sampling: Perform asynchronous quality evaluation on a certain proportion of requests through Prometheus-2 and Qwen2.5, feed the results back to the model's accuracy reputation score to form a closed-loop optimization, while avoiding additional overhead.

The project provides an OpenAI-compatible /v1/chat/completions API endpoint, allowing existing clients to migrate seamlessly; it offers high-performance asynchronous services via uvicorn, supporting multi-worker deployment to handle high concurrency; it provides adapters for vLLM, NVIDIA Dynamo, and Ray Serve, enabling easy integration with existing inference infrastructure.

Semantic LLM Router brings new ideas to the operation and maintenance of self-hosted LLM clusters. It achieves optimal resource allocation through market mechanisms and user preferences, significantly improving resource utilization of heterogeneous model clusters, reducing operational costs, and ensuring service quality. This solution integrates economic and operational research ideas, representing the development direction of LLM inference management and providing new possibilities for the sustainable development of AI systems.