LLM Routing Benchmark Tool: A Practical Solution for Optimizing Tail Latency in Inference Services

This article introduces the open-source testing platform llm-routing-bench, which aims to evaluate and optimize routing strategies for LLM inference services, focusing on solving tail latency issues. The tool provides a standardized evaluation environment that supports fair comparison of multiple routing strategies, helping developers build efficient and reliable LLM inference services. Its core value lies in simulating real workloads and collecting fine-grained metrics to provide data support for routing strategy selection and performance optimization.

With the widespread deployment of large language models (LLMs) in various applications, performance optimization of inference services has become a core challenge in engineering practice. Unlike traditional web services, LLM inference has unique computational characteristics: request processing time is highly variable, influenced by factors such as input/output length, model complexity, and batching strategies. In this context, tail latency—the response time of the slowest portion of requests—becomes a key bottleneck for user experience. Even if average latency is good, if 1% of requests take seconds or even tens of seconds to complete, the overall service availability and user satisfaction will be greatly reduced. Routing strategies are important means to control tail latency. By intelligently distributing requests to different backend instances, load can be effectively balanced, hotspots avoided, and queuing wait times reduced. However, different routing strategies perform very differently in different scenarios, and the lack of systematic evaluation tools has long been a pain point in the industry.

llm-routing-bench is a specially designed testing platform for measuring and comparing the effectiveness of various routing strategies in reducing tail latency of LLM inference services. This project provides researchers and engineers with a standardized evaluation environment, enabling fair comparison of different routing algorithms.

Core Features and Design Goals

The design of this testing platform revolves around the following core goals: Real Workload Simulation: The platform can simulate real LLM inference request patterns, including request arrival time distribution, input/output length variations, and mixing of requests with different priorities. Support for Multiple Routing Strategies: Built-in support for various classic and cutting-edge routing algorithms, including Round Robin, Least Connections, Predictive Routing, and Learning-based Routing. Fine-Grained Metric Collection: In addition to basic latency metrics, the platform collects detailed metrics such as queue length, instance utilization, and cache hit rate to help deeply understand the behavioral characteristics of routing strategies. Extensible Architecture: A modular architecture is designed, allowing users to easily add new routing strategies, customize workload patterns, or connect to different backend simulators.

Technical Implementation Points

The technical implementation of llm-routing-bench reflects an in-depth understanding of the characteristics of LLM inference services: Request Feature Modeling: The platform models the features of LLM requests in detail. The number of input tokens, output tokens, and their ratio all affect processing time, and these features are considered in routing decisions. Backend Instance Simulation: To conduct large-scale testing without a real GPU cluster, the platform implements a highly realistic backend instance simulator. The simulator can reproduce the latency distribution, batching behavior, and resource competition effects of real LLM services. Statistical Analysis Methods: Evaluating tail latency requires robust statistical methods. The platform uses various statistical techniques such as quantile analysis, empirical distribution functions, and hypothesis testing to ensure the reliability and interpretability of evaluation results.

llm-routing-bench supports evaluation of multiple types of routing strategies, each with specific applicable scenarios and trade-offs:

Static Strategies

Round Robin: The simplest strategy, distributing requests to instances in order. Advantages: simple implementation, stateless; Disadvantages: cannot handle performance differences and load imbalance between instances. Weighted Round Robin: Assigns different weights to different instances based on round robin. Suitable for heterogeneous clusters but requires manual weight tuning. Least Connections: Distributes requests to the instance with the fewest current connections. Handles scenarios with large differences in request processing time well, but responds slowly to sudden loads.

Dynamic Strategies

Shortest Queue: Distributes requests to the instance with the shortest estimated waiting time. Requires maintaining queue status information for each instance; higher implementation complexity but usually better results. Predictive Routing: Uses historical data and machine learning models to predict the processing time of a specific request on each instance, then selects the optimal instance. Requires high prediction accuracy but has significant effects under stable workloads. Learning-based Routing: Uses reinforcement learning to learn optimal routing strategies online. Adapts to workload changes but requires certain exploration costs and training time.

Hybrid Strategies

In practice, the most effective solutions are often combinations of multiple strategies. For example, combining Least Connections and Predictive Routing: use Least Connections for coarse-grained filtering, then use predictive models to select the optimal instance among candidates. llm-routing-bench supports such flexible strategy combinations and parameter tuning.

Through extensive experiments with llm-routing-bench, the project maintainers have summarized some best practices for LLM routing:

Key Insights for Tail Latency Optimization

Impact of Batching Cannot Be Ignored: LLM inference services usually use dynamic batching to improve throughput, but this introduces additional queuing delays. Routing strategies need to consider the depth of the batching queue, not just the immediate load of instances. Value of Input Length Prediction: If the input length of a request (or at least its distribution) can be accurately predicted, routing decisions can be more precise. The optimal target instances for short and long requests may be completely different. Special Challenges of Heterogeneous Clusters: When the cluster contains GPUs of different models or instances with different configurations, simple load balancing strategies often fail. More intelligent routing algorithms that consider instance capability differences are needed.

Configuration Recommendations

For deployment scenarios of different scales and characteristics, the experimental results of llm-routing-bench provide the following recommendations: Small-scale Homogeneous Clusters (fewer than 10 instances with the same configuration): The Least Connections strategy combined with appropriate timeout and retry mechanisms usually achieves good results. Large-scale Heterogeneous Clusters: It is recommended to use Learning-based Routing or Predictive Routing, and recalibrate the model regularly based on actual operation data. High-priority Mixed Workloads: Consider using multi-level queues and priority-aware routing strategies to ensure low latency for critical requests while maintaining overall throughput.

llm-routing-bench is suitable for multiple application scenarios:

Routing Algorithm Research and Development

For scholars and engineers researching new routing algorithms, llm-routing-bench provides a standardized evaluation benchmark. Researchers can focus on algorithm innovation without reinventing the wheel to build a testing environment.

Production Environment Selection Decisions

Before deploying LLM inference services, operation teams can use llm-routing-bench to simulate expected workloads, evaluate the performance of different routing strategies, and provide data support for configuration decisions in the production environment.

Performance Regression Testing

Integrating llm-routing-bench into the CI/CD process allows automatic performance regression testing after code changes, timely detecting the impact of routing logic changes on latency characteristics.

Capacity Planning and Scalability Analysis

By simulating clusters of different scales and workloads of different intensities, llm-routing-bench can help with capacity planning and determine the minimum resource configuration required to meet specific latency SLAs.

As an open-source project, llm-routing-bench has some known limitations: Gap Between Simulation and Reality: Although the backend simulator strives for realism, there are still differences from the behavior of real GPU clusters. Some edge cases (such as memory overflow, driver bugs) are difficult to reproduce in the simulation environment. Representativeness of Workloads: The workload patterns provided by the project are based on public datasets and literature, which may not fully represent the actual request distribution of specific application scenarios. Users may need to collect and import workload data more in line with their own scenarios. Support for Multimodal Inference: The current version mainly focuses on inference optimization for text LLMs, with limited support for the special needs of multimodal models (such as vision-language models).

The project maintainers stated in the roadmap that future development directions include:

• Supporting richer backend simulation options, including behavioral simulation of emerging inference engines (such as vLLM, TensorRT-LLM)
• Introducing more advanced workload generation models, supporting replay based on real logs
• Extending support for multimodal inference and streaming generation scenarios
• Developing visualization tools to help more intuitively understand routing decisions and performance bottlenecks.

llm-routing-bench provides a practical open-source tool for performance optimization of LLM inference services. Today, as LLM applications become increasingly popular, latency optimization of inference services has become a key factor affecting user experience. By systematically evaluating and comparing different routing strategies, developers can make more informed architectural decisions and build efficient and reliable LLM inference infrastructure. For any engineer who needs to optimize the performance of LLM services, llm-routing-bench is a project worth paying attention to.