AITestBench: A Practical Tool for Performance Evaluation of LLM Inference Servers

AITestBench is a lightweight performance testing tool for LLM inference servers, designed to address the problem that general-purpose pressure testing tools cannot accurately simulate the unique load patterns of LLMs. It provides multi-dimensional performance metrics, flexible test configurations, and standardized protocols to help developers and operations personnel evaluate the performance of different models and inference backends, offering data support for model selection and capacity planning in production environments.

Traditional web service pressure testing tools (such as Apache Bench and wrk) cannot accurately simulate the unique load patterns of LLM inference. LLM inference has the following characteristics:

• Variable-length output: The same input may produce outputs of vastly different lengths, leading to significant fluctuations in response time
• Streaming transmission: Modern LLM APIs often use SSE (Server-Sent Events) for streaming returns, requiring special handling to accurately measure first-token latency and full response time
• Context sensitivity: The length of the input sequence directly affects computational complexity, and there are significant differences in throughput performance between short and long prompts
• Concurrency characteristics: The concurrency handling capability of GPU inference differs from CPU services; simply increasing the number of concurrent requests does not necessarily lead to linear throughput improvement These factors make it difficult for general-purpose tools to provide valuable performance data in LLM scenarios.

Multi-Dimensional Performance Metrics

It can measure key metrics such as Time to First Token (TTFT), throughput, end-to-end latency, and concurrency performance, forming a complete performance profile.

Flexible Test Configuration

Supports modes like fixed concurrency testing, progressive pressure application, custom prompts, and comparison of different models, which are close to real application scenarios.

Standardized Testing Protocol

Follows the OpenAI-compatible API format, allowing testing of commercial LLM services (e.g., OpenAI), open-source inference engines (e.g., vLLM, TensorRT-LLM), and self-hosted model services, facilitating comparison between different solutions.

Typical use cases of AITestBench include:

• Model Selection Decision: Provides objective performance data support, e.g., assisting in evaluating inference efficiency when choosing between Llama-3-8B and Qwen-7B
• Inference Backend Optimization Verification: Verifies the effects of adjusting batch size, quantization schemes, or upgrading inference engines
• Capacity Planning and SLA Formulation: Finds performance inflection points through progressive pressure application, providing a basis for production capacity planning and SLA commitments
• Continuous Performance Monitoring: Integrates into CI/CD pipelines to automatically run performance regression tests and detect performance degradation in a timely manner

To obtain meaningful test results, it is recommended to follow these practices:

1. Use real prompts: Reflect actual business scenarios, including typical input length distributions
2. Focus on P99 latency: Avoid being misled by average values; understand long-tail latency to reflect real user experience
3. Warm-up tests: GPU inference services need to be warmed up to reach a stable state
4. Multiple sampling: Due to the randomness of LLM outputs, single test results may fluctuate significantly; it is recommended to take multiple samples and average them
5. Monitor resource usage: Combine with metrics such as GPU utilization and memory occupancy to fully understand system bottlenecks

Compared to complex benchmark suites (e.g., lm-evaluation-harness), AITestBench is concise and focused, only targeting inference performance measurement, with a low learning and usage threshold. Compared to commercial APM tools, it is open-source and free, and can be flexibly integrated into automated workflows.

In the process of LLM applications moving from prototype to production, performance evaluation is indispensable. AITestBench fills the tool gap with its concise and practical design, and it is worth adding to your toolbox whether for model selection, inference backend optimization, or capacity planning.