inference-bench: A Fair Showdown of Large Model Inference Engines

The open-source project inference-bench provides a fair comparison benchmark for three mainstream inference engines: vLLM, SGLang, and llama.cpp. It comprehensively tests key metrics such as throughput, latency, and success rate on a single L4 GPU, aiming to resolve information asymmetry in large model inference engine selection and provide reliable data support for production environment choices.

With the widespread application of large language models, efficient deployment has become a core challenge. There are many inference engines on the market (e.g., vLLM, SGLang, llama.cpp), but official benchmark test conditions vary, community reports are inconsistent, and there is a lack of standardized evaluation, leading to confusion in selection. The inference-bench project addresses this dilemma through a reproducible, standardized testing framework that allows different engines to compete fairly under the same hardware and load conditions.

inference-bench selects three representative open-source inference engines:

vLLM: Developed by Berkeley, its core is the PagedAttention technology, which supports continuous batching to improve GPU memory utilization and throughput.

SGLang: Focuses on structured generation and multimodality, introducing the RadixAttention mechanism to accelerate multi-turn conversations and providing a flexible programming interface.

llama.cpp: A star in the GGML ecosystem, it features extreme CPU inference optimization, supports multiple quantization schemes, and also provides a CUDA backend.

The tests are conducted on a single NVIDIA L4 GPU (24GB VRAM, a common production configuration). Evaluation metrics include throughput, Time to First Token (TTFT), Time per Output Token (TPOT), tail latency, and success rate. The tests cover two workloads: short prompt with short output (chat applications) and long prompt with long output (document generation).

The test results show the pros and cons of each engine:

• vLLM: Leads in throughput; its mature PagedAttention and continuous batching have obvious advantages under high concurrency.

• SGLang: Excels in structured generation and multi-turn conversation scenarios; RadixAttention reduces TTFT, making it suitable for format-required scenarios like JSON generation.

• llama.cpp: Its absolute throughput is lower than GPU engines, but its quantization support and cross-platform capabilities make it suitable for resource-constrained environments.

Significant differences exist under different workloads: gaps are small at low concurrency, but architectural differences become apparent at high concurrency.

Selection recommendations based on test results:

• For high throughput, low latency, and standard model scenarios: Choose vLLM, which has a mature ecosystem and low maintenance costs.

• For structured generation, multi-turn conversations, or flexible control logic: Choose SGLang; prefix caching improves conversation performance.

• For resource-constrained environments or quantization needs: Choose llama.cpp, which has a rich GGUF format ecosystem.

The best choice requires actual testing in the specific scenario, and inference-bench provides a standardized framework for support.

The value of inference-bench: Establishes an open and reproducible evaluation benchmark that anyone can reproduce and verify; its modular design makes it easy to expand new engines and scenarios; it provides performance analysis tools and visualization scripts to improve transparency. The project promotes the healthy development of the field and is expected to become a community-standard testing platform.