RouteLLM-rs: A Distributed LLM Inference Routing System Based on Consistent Hashing

RouteLLM-rs is a distributed LLM inference router written in Rust. Its core uses the consistent hashing algorithm to achieve efficient request distribution and load balancing. It addresses the shortcomings of traditional load balancing in LLM scenarios, improves cache utilization through request affinity, supports smooth scaling, has high concurrency processing capabilities, and provides a professional routing solution for building scalable LLM inference services.

With the widespread application of large language models (LLMs) in production environments, a single inference node often cannot meet the demands of high-concurrency scenarios. Enterprises usually need to deploy multiple inference instances to share the load, but how to efficiently route user requests to the appropriate backend nodes has become a key technical challenge.

Traditional load balancing methods such as round-robin or random selection are simple to implement but have obvious flaws in LLM inference scenarios: the same or similar requests may be distributed to different nodes, leading to low cache hit rates; the execution time of long text generation tasks varies greatly, which easily causes uneven load among nodes; session context needs to be consistent across multiple requests, putting higher requirements on routing strategies.

Consistent hashing is a special distributed hashing scheme originally proposed by David Karger et al. from MIT to solve hot spot problems in distributed cache systems. Unlike the traditional hash modulo method, it organizes the hash space into a ring structure, where both nodes and request keys are mapped to a position on the ring.

RouteLLM-rs applies consistent hashing to LLM inference routing, bringing three major advantages:

1. Request Affinity: Based on the hash value of the request content, similar or identical prompts are routed to the same backend node, effectively reusing the node's local cache (such as KV Cache) and reducing repeated computation overhead;
2. Smooth Scaling: When adding or removing backend nodes, only a small portion of adjacent requests on the ring need to be rerouted, and most mapping relationships remain unchanged, which is suitable for elastic inference services;
3. Load Balancing: By assigning multiple virtual nodes to each physical node, finer-grained load distribution is achieved, avoiding individual nodes becoming hot spots.

Comparison with other routing schemes:

Feature	Simple Round-Robin	Least Connections	RouteLLM-rs
Implementation Complexity	Low	Medium	Medium
Cache Friendliness	Poor	Poor	Excellent
Scaling Smoothness	Poor	Medium	Excellent
Session Persistence	Requires additional handling	Requires additional handling	Natively supported
Performance Overhead	Extremely low	Low	Low

RouteLLM-rs's architecture design reflects the advantages of Rust in systems programming:

Request Reception Layer

The system exposes an interface compatible with the OpenAI API, receives client inference requests, and can be seamlessly integrated into the existing LLM application ecosystem as a transparent proxy.

Routing Decision Layer

After receiving a request, it extracts key features (such as model name, prompt content, parameter configuration, etc.) to calculate the hash value, and locates the target backend node on the consistent hash ring. Decision considerations include:

• Node Health Status: Regular health checks to automatically exclude faulty nodes;
• Current Load: Real-time monitoring of the number of concurrent requests and processing delays of each node;
• Cache Affinity: Prefer nodes that may have relevant caches.

Backend Connection Pool

Maintains a persistent connection pool with each backend inference node, avoiding the overhead of establishing a new connection for each request, and supports HTTP/2 multiplexing to improve throughput efficiency.

Response Handling and Monitoring

Responses are returned to the client in a streaming manner, while detailed metrics (routing decision time, backend processing delay, cache hit status, etc.) are recorded to provide data support for operation and maintenance optimization.

Deployment configuration uses TOML format, and typical configurations include:

• Backend Node List: Specify available inference service addresses and weights;
• Hash Strategy: Select hash algorithms (such as MurmurHash3, CityHash) and the number of virtual nodes;
• Health Check Parameters: Define check intervals, timeout periods, and failure thresholds;
• Cache Configuration: Enable/disable request/response caching, set cache size and expiration policies;
• Monitoring Endpoint: Configure the Prometheus metrics exposure port.

Choosing Rust as the implementation language brings unique technical advantages to RouteLLM-rs:

• Zero-Cost Abstraction: Rust's high-level abstractions are optimized at compile time, so complex hash calculations and routing logic have no runtime overhead;
• Memory Safety: The ownership system eliminates the risks of memory leaks and wild pointers, which is crucial for long-running infrastructure services;
• Asynchronous High Performance: Based on Tokio's asynchronous runtime, a single thread can handle a large number of concurrent connections, and CPU utilization is significantly better than traditional multi-threaded models;
• Compile-Time Optimization: Generics and const generics allow configuration parameters (such as the number of virtual nodes, hash ring size) to be determined at compile time, generating highly optimized machine code.

In typical production environments, RouteLLM-rs demonstrates excellent performance characteristics:

• Routing Delay: Sub-millisecond routing decision time, with minimal impact on end-to-end latency;
• Throughput: A single instance can support tens of thousands of routing decisions per second;
• Cache Hit Rate: In scenarios with many similar requests, the cache hit rate can reach 60-80%.

Optimization recommendations: Adjust the number of virtual nodes according to the actual request pattern, set the health check frequency reasonably to balance timeliness and overhead, and monitor the P99 latency differences between nodes to identify potential problems.

RouteLLM-rs is particularly suitable for the following scenarios:

• Batch processing workloads that need to handle a large number of similar requests;
• Interactive applications that are latency-sensitive and where cache hit rate has a significant impact;
• Elastic inference services that require frequent scaling;
• Multi-tenant environments that need request isolation.

However, for scenarios where request content is highly random and there are almost no repeated patterns, the advantages of consistent hashing may not be manifested, and simple load balancing strategies may be more appropriate.

RouteLLM-rs represents the trend of LLM infrastructure moving towards more specialization and high performance. By combining mature technologies in the distributed systems field (consistent hashing) with Rust's system-level performance advantages, it provides a solid routing layer solution for building scalable and efficient LLM inference services. For teams planning or optimizing LLM inference architectures, RouteLLM-rs is worth a careful evaluation.

Future development directions:

• Intelligent Prefetching: Based on request pattern prediction, preload the model weights that may be needed into specific nodes in advance;
• Multi-Model Routing: Support finer-grained routing decisions based on model type and version;
• Federated Learning Integration: Support the coordination and distribution of model updates at the routing layer;
• Edge Inference Support: Extend to edge computing scenarios to achieve central-edge collaborative inference.