llm-inference-benchmarks: A Benchmark Toolset for LLM Inference Performance

This is an open-source project focused on evaluating the inference performance of large language models (LLMs). It provides a standardized testing framework and tools to assess the performance of different models, hardware configurations, and inference engines. Its core value lies in helping developers objectively compare inference performance under different configurations, providing data support for model selection, hardware procurement, engine optimization, and capacity planning, and promoting reproducible research in the field of LLM inference optimization.

LLM inference performance is affected by multiple factors: model architecture (Transformer variants, MoE architecture, quantization strategies), hardware platforms (GPU models, VRAM capacity, CPU/GPU collaboration), inference engines (vLLM, TensorRT-LLM, llama.cpp, TGI, etc.), and optimization techniques (KV Cache management, Continuous Batching, Speculative Decoding). Without a unified benchmark, performance comparisons often become invalid "apple-to-orange" comparisons.

The toolset covers the following core testing dimensions:

Throughput Testing

Measures the number of tokens or requests processed per unit time. Key metrics include tokens per second (tok/s), total throughput (req/s), and Time To First Token (TTFT).

Latency Testing

Focuses on single-request response speed, including end-to-end latency, per-token latency, and P50/P99 percentiles.

Resource Utilization

Monitors hardware consumption: VRAM usage, GPU utilization, power consumption, and energy efficiency.

Accuracy Comparison

Verifies the perplexity changes of quantized models and the accuracy of downstream tasks.

High-quality benchmarking follows four key principles:

1. Standardized Input: Use representative datasets (ShareGPT, LongBench, synthetic loads).
2. Warm-up and Stabilization: Eliminate interference from cold starts and cache misses.
3. Multiple Sampling: Repeat tests and report statistical distributions.
4. Controlled Variables: Change only one variable (model/engine/hardware) at a time to ensure comparability.

This toolset has direct value in the following scenarios:

• Model Selection: Objectively compare the throughput and latency of models like Qwen2.5-72B and Llama-3.1-70B.
• Hardware Procurement: Evaluate the cost-effectiveness of A100 vs H100 vs RTX4090.
• Engine Optimization: Compare the optimization effects of vLLM's PagedAttention and TensorRT-LLM.
• Capacity Planning: Derive the required GPU quantity configuration based on target QPS and latency SLA.

The emergence of the llm-inference-benchmarks project reflects the evolution of LLM engineering from "usable" to "user-friendly". As inference optimization technologies develop rapidly, standardized and reproducible benchmarking will become the infrastructure for community collaboration.