LLMTest-Perf: An Automated Solution for LLM Inference Performance Regression Testing

LLMTest-Perf is an open-source tool dedicated to performance testing of large language model (LLM) inference. It aims to help development teams automatically detect performance regression issues in metrics such as latency, throughput, and Time to First Token (TTFT) before release. Designed for the unique characteristics of LLM inference, it supports multi-dimensional performance evaluation, automated regression detection, CI/CD integration, and compatibility with mainstream inference engines, filling the gap in performance testing within the LLM engineering toolchain.

LLM inference performance testing differs fundamentally from traditional software testing: it involves memory-intensive attention computation and compute-intensive forward propagation, with performance influenced by multiple factors such as model architecture, parameter size, sequence length, batch size, and hardware configuration. Its iterative generation mode requires evaluating multi-dimensional metrics like TTFT (user-perceived latency) and throughput (system processing capacity). Manual testing is time-consuming and lacks consistency, while general-purpose tools fail to capture LLM-specific metrics, posing challenges for performance regression validation in continuous iterative development.

LLMTest-Perf is built specifically for LLM inference performance testing, with the core goal of establishing an automated performance regression testing workflow. Unlike general-purpose benchmarking tools, it deeply understands the characteristics of LLM inference, providing targeted metrics (TTFT, TPOT, end-to-end latency, performance stability, etc.) and evaluation methods, focusing on solving performance regression issues in LLM scenarios.

1. Latency Testing: Measures TTFT (Time to First Token, from request to first token return), TPOT (Time per Output Token, average time per output token), and end-to-end latency to help understand user experience;
2. Throughput Testing: Evaluates tokens/second metrics under different batch sizes and concurrent requests to detect performance jitter or degradation;
3. Regression Detection: Establishes a performance baseline, automatically compares current performance with the baseline, issues alerts, and provides detailed comparison reports (e.g., metric degradation magnitude, possible causes).

Request Modes: Supports fixed-length testing, variable-length testing (simulating real-world randomness), and real dataset testing; Load Modes: Constant rate testing, burst load testing (simulating traffic peaks), and progressive pressure testing (until system saturation); Long Context Testing: Generates input sequences of different lengths to evaluate the impact of KV cache management on performance.

LLMTest-Perf supports command-line interfaces and configuration files, enabling seamless integration into mainstream CI platforms like GitHub Actions, GitLab CI, and Jenkins. It can run tests during the Pull Request phase, using results as a reference for code reviews; and perform comprehensive performance regression validation before release. Test results can generate HTML reports (including trend charts, metric comparisons, regression summaries) that are automatically uploaded or sent to team channels.

Compatibility: Supports mainstream inference engines like vLLM, TensorRT-LLM, llama.cpp, and TGI via OpenAI-compatible APIs; provides adaptation interfaces for self-developed engines; can evaluate the benefits of optimization techniques such as quantization, KV cache optimization, continuous batching, and speculative decoding; Application Cases: Model version upgrade validation, inference engine migration evaluation, hardware selection decision-making, performance optimization iteration (data-driven optimization workflow).

Limitations: Performance testing consumes computing resources; resource-constrained environments need to balance coverage and consumption; LLM performance is affected by factors like hardware temperature and system load, making it difficult to completely eliminate test noise (mitigated via multiple sampling and statistical testing); Future Directions: Support performance testing for multimodal models, add energy efficiency metrics, intelligent regression root cause analysis, and establish a community-shared performance baseline database.