llm_speedtest: A Local Large Language Model Inference Performance Testing Tool

llm_speedtest is an open-source tool focused on inference performance testing of local large language models (LLMs), designed to help users quantitatively evaluate the inference speed, throughput, latency, and memory usage of locally deployed LLMs. As the demand for local deployment grows, accurately assessing performance across different models and hardware configurations has become a practical issue. This tool strikes a balance between simplicity and professionalism, providing users with a standardized testing solution.

LLM performance evaluation is complex, involving multi-dimensional metrics such as generation speed (Tokens/Second), first-token latency, throughput, and memory usage, which are affected by factors like model architecture, quantization precision, and hardware type. Existing solutions have limitations: general benchmarks (e.g., MLPerf) are too complex; framework-built tools (e.g., llama.cpp's benchmark) only support specific frameworks; manual scripts make cross-comparison difficult. Therefore, a specialized tool is needed to address these issues.

The core features of llm_speedtest include:

1. Standardized testing process: Preheating phase to eliminate cold start effects, multi-round tests to take average values, multi-dimensional sampling of key metrics;
2. Flexible configuration: Supports adjusting parameters such as input length, output length, concurrency count, and number of test rounds;
3. Clear output report: Structured results facilitate quick overview, export analysis, and cross-comparison. The design pursues a balance between simplicity and professionalism.

This tool is suitable for multiple scenarios:

1. Hardware selection decision: Quantitatively evaluate the support level of different hardware for the target model;
2. Model optimization verification: Verify performance improvements after optimizations like quantization and pruning;
3. Deployment scheme comparison: Compare the performance of the same model under different deployment methods (e.g., llama.cpp, vLLM);
4. Performance regression detection: Add tests to CI pipelines to promptly detect the impact of code changes on performance.

In terms of technical implementation:

• Integration with inference engines: Supports OpenAI-compatible APIs, local process calls (e.g., llama.cpp, ollama), and Python bindings (e.g., transformers, vLLM);
• Measurement accuracy considerations: Uses high-precision timers, distinguishes between actual generation and waiting time, records system load, monitors hardware throttling, etc., to ensure accurate results.

Recommendations for use:

1. Test environment preparation: Close irrelevant programs, connect to power (for laptops), ensure good heat dissipation, take averages from multiple tests;
2. Result interpretation: Comprehensive evaluation combining model size, quantization precision, and hardware cost; pay attention to latency percentiles (P50/P95/P99); control variables when comparing (same quantization method, prompt length, etc.).

Current limitations: Dependence on external inference backends, limited platform compatibility, insufficient coverage of test scenarios. Future improvement directions: Built-in support for common inference backends, generating visual reports, establishing a community performance database, adding stress testing mode, etc.

llm_speedtest represents an important direction in the tooling of the LLM ecosystem—moving from 'usable' to 'easy to use', and from 'roughly knowing' to 'precise quantification'. As the user base for local deployment expands, such tools will play an increasingly important role and serve as a good starting point for building a reliable performance testing toolbox.