xk6-llm: A Professional Load Testing Tool for LLM Inference Services

In the implementation of LLM applications, inference service performance directly affects user experience and operational costs. xk6-llm is an LLM-specific load testing framework extended from k6, supporting measurement of key metrics such as TTFT, ITL, TPOT, compatible with OpenAI API standards, and integrable with Prometheus and Grafana monitoring systems, addressing the pain points where traditional tools fail to meet LLM inference scenarios.

Traditional HTTP load testing tools (e.g., k6, JMeter) can only measure throughput and latency, and cannot cover special dimensions of LLM inference such as streaming output and first-token latency. xk6-llm inherits k6's high performance and ease of use, adds professional metric collection capabilities for LLM scenarios, supports all inference servers compatible with OpenAI API, and has wide applicability.

xk6-llm provides four core metrics:

1. TTFT (Time to First Token): The time from request to the first token, affecting user response perception;
2. ITL (Inter-Token Latency): Streaming generation speed, affecting output fluency;
3. TPOT (Time per Token): Averages time per token, integrating factors like model computation, which is the core of optimization;
4. Goodput: Actual token generation rate, reflecting real service capability.

xk6-llm innovatively introduces cost and energy consumption dimensions:

• Cost metrics: Calculate inference costs based on token usage to evaluate the economic efficiency of model configurations;
• Energy consumption metrics: Measure inference energy consumption to support green AI and sustainable operations. These metrics combine performance testing with business value and operational costs.

xk6-llm natively integrates with Prometheus and Grafana:

1. Historical data tracking: Long-term storage of results to track changes from optimizations and upgrades;
2. Visual analysis: Grafana dashboards display metric trends;
3. Alert mechanism: Timely notifications when performance degrades;
4. CI/CD integration: Automate performance regression testing.

xk6-llm is suitable for:

• Model selection evaluation: Compare hardware performance of different models;
• Inference optimization verification: Validate the effects of solutions like vLLM and TensorRT-LLM;
• Capacity planning: Determine GPU resources needed to support concurrency;
• Performance regression testing: Ensure performance does not degrade after model updates;
• Vendor comparison: Evaluate differences in LLM APIs from cloud service providers.

xk6-llm is developed in Go language, using k6's extension mechanism to support LLM-specific protocols. By parsing OpenAI API streaming responses, it accurately calculates token arrival times, ensuring test accuracy and high tool performance.

xk6-llm fills the tool gap in LLM inference performance testing, providing AI teams with professional and comprehensive testing methods. As LLM applications become widespread, high-performance and low-cost services require scientific testing methods, and xk6-llm is worth including in the toolchain.