llm-grill: A One-Stop Performance Benchmarking Tool for LLM Inference Servers

llm-grill is a command-line tool specifically designed for performance benchmarking of mainstream LLM inference servers. It supports multiple backends including vLLM, SGLang, llama.cpp, and LiteLLM, helping developers quickly evaluate and compare the performance of different inference solutions, and addressing the pain point of time-consuming and labor-intensive manual testing in LLM deployment.

In LLM deployment practice, choosing the right inference server is a critical decision. Different inference frameworks vary in performance aspects such as throughput, latency, and memory usage, while manual testing and comparison of these solutions are often time-consuming and labor-intensive. The llm-grill project was born to address this pain point, providing unified and standardized performance benchmarking.

llm-grill currently supports four mainstream LLM inference backends:

• vLLM: A GPU inference engine developed by UC Berkeley, with PagedAttention algorithm at its core, improving GPU memory utilization and concurrent throughput, suitable for production environments;
• SGLang: A structured generation language with an efficient inference runtime, excelling at handling structured outputs (e.g., JSON schema);
• llama.cpp: A C++ implementation supporting consumer-grade hardware and multiple quantization formats (GGUF), suitable for local deployment and edge computing;
• LiteLLM: A unified API gateway supporting over 100 model providers, enabling performance testing of remote services.

Unified Testing Interface

Regardless of the underlying inference server used, users can test with the same command parameters, eliminating learning costs.

Key Performance Metrics

Collects and reports metrics such as throughput (tokens per second), time to first token (TTFT), end-to-end latency, and concurrent processing capability.

Scenario-Based Testing

Supports simulating chat scenarios (focusing on TTFT), batch processing scenarios (high concurrent throughput), and long text generation (stability evaluation).

Architecture Selection Decision

Provides objective data support to help balance choices such as vLLM's high throughput vs. llama.cpp's flexibility;

Performance Regression Testing

Establishes performance baselines when upgrading versions or replacing hardware to avoid performance degradation;

Capacity Planning

Determines single-node concurrency to provide a basis for cluster scaling;

Vendor Comparison

Connects to multiple service providers via LiteLLM to objectively compare response speeds of different cloud service providers.

llm-grill follows the Unix philosophy (do one thing well). It communicates with each inference server via standardized HTTP interfaces, uses asynchronous IO to generate high-concurrency requests, and applies statistical methods to calculate stable performance metrics. Outputs include raw data (CSV/JSON), visual charts (latency distribution, throughput trends), and summary reports (average latency, P99 latency, throughput, etc.).

The emergence of llm-grill reflects the evolution of the LLM ecosystem from "usable" to "user-friendly". As inference engines become more diverse, the community needs standardized evaluation methods, and this tool fills the gap by providing developers with an objective basis for selection.

llm-grill is a practical LLM inference performance testing tool that supports multiple backends via a unified interface, providing data support for architecture selection, performance optimization, capacity planning, etc. It is recommended that teams building or optimizing LLM services add it to their toolchain.