llm-inference-bench: LLM Inference Performance Benchmark Tool with Visualization Panel

This article introduces the open-source tool llm-inference-bench, which supports two major inference engines—SGLang and vLLM—and provides a Rich TUI visualization panel to measure token generation speed under different concurrency levels and context lengths. The tool aims to help developers and operation teams conduct LLM inference performance tests, providing data support for capacity planning, engine selection, and performance tuning.

As the deployment scale of LLMs in production environments expands, inference performance optimization has become a core challenge. Traditional tests only focus on simple throughput, but in real scenarios, factors such as the number of concurrent users, input context length, output token count, and model quantization methods all affect inference latency and throughput. The lack of systematic tools makes it difficult to accurately plan capacity and perform tuning.

llm-inference-bench natively supports two major engines: SGLang (developed by Berkeley, with high throughput and flexibility) and vLLM (widely used in the community, with memory optimized via PagedAttention), allowing comparison of backend performance under the same conditions. Testing dimensions include: concurrency level (simulating multi-user requests), context length (from short text to long documents), and decoding throughput (token generation speed, the main source of user-perceived latency).

The tool includes core components: llm_decode_bench.py (benchmark logic, interacting with engines, collecting data, calculating metrics), llm_cjk_watchdog.py (monitoring CJK character processing to ensure multilingual accuracy), tools/ (auxiliary scripts such as data post-processing), and docs/ (usage guides). It adopts a modular design, making it easy to extend new engines or metrics.

The tool is suitable for: 1. Capacity planning (determine the maximum number of concurrent users for hardware and find performance saturation points); 2. Performance regression testing (run automatically in CI pipelines to compare with historical baselines); 3. Engine selection (fairly compare throughput, memory usage, etc., between SGLang and vLLM); 4. Tuning validation (verify optimization effects such as quantization and batch size adjustment).

Comparison with similar tools: vLLM official benchmark (vLLM only), SGLang benchmark (SGLang only), llmperf (general framework). The advantages of llm-inference-bench lie in its unified multi-engine support and intuitive Rich TUI interface, which lowers the threshold for cross-engine comparison.

llm-inference-bench fills the gap in LLM inference benchmarking, reduces the usage threshold through its visualization interface, and helps teams avoid resource waste or service degradation. It is recommended that teams deploying LLM services include it in their evaluation list. Future plans include supporting more engines (such as TensorRT-LLM, llama.cpp) and report formats (HTML, JSON, CSV).