llm-inference: Comprehensive LLM Inference Performance Evaluation Tool & One-Click Deployment Solution

llm-inference is an open-source tool maintained by Yoannoza (hosted on GitHub) that measures key LLM inference metrics (TTFT, TPOT, throughput, cost, VRAM usage). It supports any OpenAI-compatible API and includes the infer-serve feature for one-click deployment of GGUF models via llama.cpp. This thread breaks down its background, core functions, usage, and value.

With the popularization of LLM applications, inference performance directly impacts user experience and cost-effectiveness. However, developers lack systematic tools to evaluate performance across different API providers, model versions, and hardware configurations. llm-inference addresses this pain point by providing a complete evaluation toolkit to help make informed technical decisions.

The tool focuses on 5 key metrics:

• TTFT: Time to first token (affects interactive app response speed; influenced by model loading, input processing, network delay, queue wait time)
• TPOT: Time per output token (determines output fluency)
• Throughput: Token/request count per unit time (measures scalability)
• Cost: Per request or per 1k token expense (critical for commercial apps)
• VRAM Usage: Resource consumption (helps choose hardware configurations)

llm-inference offers a command-line interface with features:

• Basic performance testing: Auto-run test cases to collect metrics
• Concurrent pressure testing: Simulate multi-user scenarios to find bottlenecks
• Long-term stability testing: Detect memory leaks or performance decay
• Custom config: Define input/output length, concurrent requests, test duration, sampling temperature, etc.

The infer-serve function simplifies local deployment:

• GGUF Support: Uses llama.cpp to run GGUF quantized models (balances precision and resource needs)
• One Command Deployment: No complex config/dependencies—single command starts a full inference service
• OpenAI Compatible: Deployed service uses OpenAI-like API, allowing seamless app integration without client code changes

Key use cases:

• API Selection: Compare commercial API providers with real data
• Model Optimization: Validate effects of quantization/pruning
• Capacity Planning: Pressure tests to determine system limits for infrastructure planning
• Performance Regression: Integrate into CI/CD to catch regressions from model updates
• Local Prototyping: Use infer-serve for quick local environment setup for development

• Test Env Prep: Ensure stable network; close resource-heavy apps for local tests
• Test Case Design: Cover typical scenarios (short input/long output, long input/short output, multi-turn dialogues)
• Result Interpretation: Look at distribution metrics (P50, P95, P99) instead of just averages; consider multiple indicators
• Continuous Monitoring: Regular benchmark tests to track performance trends and detect anomalies

llm-inference provides a practical, comprehensive solution for LLM inference performance evaluation. Its one-click deployment lowers local inference barriers, enabling data privacy while using AI. As quantization tech and hardware improve, local deployment will become more viable. For developers needing to evaluate or deploy LLM inference services, this is a valuable open-source tool to add to their toolkit.