LLM Inference Bench：跨平台大模型推理性能基准测试工具

LLM Inference Bench is a platform-agnostic benchmark framework for LLM inference endpoints. It supports OpenAI-compatible APIs (e.g., vLLM, SGLang, TensorRT-LLM) and measures core metrics like TTFT, throughput, and failure rate. Key features include data-driven configuration recommendations, production scenario simulation, and easy-to-use CLI. It helps with inference engine selection, hardware procurement, parameter tuning, capacity planning, and performance regression testing.

As LLMs move to production, organizations face challenges in客观评估推理方案性能. Existing issues:

1. Vendor self-test data is idealized and not realistic.
2. Manual random tests lack statistical significance.
3. Focus on single metrics (e.g., only throughput) ignores trade-offs.
4. Test tools are platform-locked, hindering cross-solution comparison.
5. Parameter tuning relies on experience rather than data. These call for a standardized, cross-platform, multi-dimensional benchmark tool.

LLM Inference Bench is designed to solve the above pain points. Its core定位: platform-agnostic benchmark framework for LLM inference endpoints. Design goals: cross-platform compatibility, multi-dimensional measurement, data-driven config, production scenario simulation. Core features:

• Supports OpenAI-compatible APIs (vLLM, SGLang, etc.).
• Measures TTFT, throughput, failure rate.
• Provides vLLM configuration recommendations.
• Easy-to-use CLI interface.

The tool measures three core metrics:

1. TTFT: Time from request to first token, affecting user experience. Factors: model loading, input preprocessing, network delay. Optimization: prefix caching, tokenization speed.
2. Throughput: Tokens processed per second (output, total, request throughput). Factors: GPU capacity, batch efficiency, concurrency.
3. Failure Rate: Proportion of failed requests (timeout, OOM, connection errors). Critical for production reliability.

The tool achieves cross-platform support via:

1. OpenAI-compatible API: Uses /v1/completions endpoint (supported by vLLM, SGLang, TensorRT-LLM, Baseten, RHOAI, etc.).
2. Unified measurement: Same request format, timing method, and stats calculation across platforms.
3. Config abstraction: Users only need to provide API URL, auth info, model name; tool handles platform-specific details.

To mimic production environments, the tool supports:

1. Concurrent pressure test: Simulate multiple users with configurable concurrency, total requests, and arrival mode.
2. Variable input/output tests: Test performance with different input/output lengths to evaluate KV cache efficiency and stability.
3. Mixed workload: Combine short/long input/output tasks (e.g., simple QA, summary, creation) to reflect real usage.

The tool provides vLLM configuration recommendations based on实测 data:

• Tensor Parallelism: Recommend parallelism based on GPU count and model size.
• Batch Size: Balance delay and throughput considering memory limits.
• Scheduling Strategy: Optimize continuous batching for GPU utilization.
• KV Cache: Recommend cache size and eviction policy. Recommendations are validated against hardware constraints and vLLM's internal features.

LLM Inference Bench fills a gap in LLM inference performance evaluation. Its value:

• Ops teams: Objective assessment, bottleneck identification, capacity planning.
• Dev teams: Optimization guidance, regression protection.
• Decision-makers: Data-driven选型, ROI evaluation. Limitations: config recommendations focus on vLLM; test data may not fully represent real workloads. Usage tips: use real data for calibration, run multiple tests, combine with production monitoring.