LLM Inference Bench: Cross-Platform LLM Inference Performance Benchmark Tool

LLM Inference Bench is a platform-agnostic benchmark framework for LLM inference endpoints. It supports OpenAI-compatible APIs (e.g., vLLM, SGLang, TensorRT-LLM) and measures core metrics like TTFT, throughput, and failure rate. Key features include data-driven configuration recommendations, production scenario simulation, and easy-to-use CLI. It helps with inference engine selection, hardware procurement, parameter tuning, capacity planning, and performance regression testing.

As LLMs move to production, organizations face challenges in objectively evaluating the performance of inference solutions. Existing issues:

1. Vendor self-test data is idealized and not realistic.
2. Manual random tests lack statistical significance.
3. Focus on single metrics (e.g., only throughput) ignores trade-offs.
4. Test tools are platform-locked, hindering cross-solution comparison.
5. Parameter tuning relies on experience rather than data. These call for a standardized, cross-platform, multi-dimensional benchmark tool.

LLM Inference Bench is designed to solve the above pain points. Its core positioning: platform-agnostic benchmark framework for LLM inference endpoints. Design goals: cross-platform compatibility, multi-dimensional measurement, data-driven config, production scenario simulation. Core features:

• Supports OpenAI-compatible APIs (vLLM, SGLang, etc.).
• Measures TTFT, throughput, failure rate.
• Provides vLLM configuration recommendations.
• Easy-to-use CLI interface.

The tool measures three core metrics:

1. TTFT: Time from request to first token, affecting user experience. Factors: model loading, input preprocessing, network delay. Optimization: prefix caching, tokenization speed.
2. Throughput: Tokens processed per second (output, total, request throughput). Factors: GPU capacity, batch efficiency, concurrency.
3. Failure Rate: Proportion of failed requests (timeout, OOM, connection errors). Critical for production reliability.

The tool achieves cross-platform support via:

1. OpenAI-compatible API: Uses /v1/completions endpoint (supported by vLLM, SGLang, TensorRT-LLM, Baseten, RHOAI, etc.).
2. Unified measurement: Same request format, timing method, and stats calculation across platforms.
3. Config abstraction: Users only need to provide API URL, auth info, model name; tool handles platform-specific details.

To mimic production environments, the tool supports:

1. Concurrent pressure test: Simulate multiple users with configurable concurrency, total requests, and arrival mode.
2. Variable input/output tests: Test performance with different input/output lengths to evaluate KV cache efficiency and stability.
3. Mixed workload: Combine short/long input/output tasks (e.g., simple QA, summary, creation) to reflect real usage.

The tool provides vLLM configuration recommendations based on actual measurement data:

• Tensor Parallelism: Recommend parallelism based on GPU count and model size.
• Batch Size: Balance delay and throughput considering memory limits.
• Scheduling Strategy: Optimize continuous batching for GPU utilization.
• KV Cache: Recommend cache size and eviction policy. Recommendations are validated against hardware constraints and vLLM's internal features.

LLM Inference Bench fills a gap in LLM inference performance evaluation. Its value:

• Ops teams: Objective assessment, bottleneck identification, capacity planning.
• Dev teams: Optimization guidance, regression protection.
• Decision-makers: Data-driven selection of solutions, ROI evaluation. Limitations: config recommendations focus on vLLM; test data may not fully represent real workloads. Usage tips: use real data for calibration, run multiple tests, combine with production monitoring.