how-fast: A Precise Benchmarking Tool for LLM Inference Performance

how-fast is an open-source tool focused on in-depth measurement of LLM inference performance. It supports latency, throughput, GPU utilization monitoring, and gateway overhead isolation analysis, helping developers accurately identify system bottlenecks. It fills a critical gap in LLM inference performance testing tools and provides real data support for optimizing model services.

Traditional HTTP stress testing tools (such as wrk, ab) cannot distinguish between the first token latency (TTFT) unique to LLM inference and the full response latency, nor can they monitor GPU utilization or isolate performance losses between the gateway layer and the inference engine. In actual production, slow requests may stem from multiple links such as load balancing, gateways, inference engines, or GPU contention. Lack of fine-grained measurement leads to blind optimization.

The core concept of how-fast is isolation: it quantifies gateway layer overhead by comparing latency differences through dual-path testing (gateway path vs. direct connection path); the built-in gpu_monitor.py collects GPU utilization and memory data without additional file copying. It supports two load modes: concurrency mode (N parallel threads to find the throughput upper limit) and QPS mode (Poisson-distributed request arrival to test SLO compliance under real traffic).

how-fast provides a complete CLI workflow: define experiments (YAML configuration for model, GPU type, etc.) → generate startup scripts → deploy to GPU servers → verify connectivity → execute benchmark tests → measure gateway overhead. It supports parameter scanning (sweep command) to find latency-throughput inflection points, and SLO validation (define thresholds via slos.yaml to automatically generate compliance reports).

Each test generates files such as requests.csv (request details), gpu_metrics.csv (GPU data), summary.json (aggregated metrics), and slo_report.json (compliance status). The project architecture is lightweight and easy to extend, using Python asynchronous IO for high concurrency, Pydantic for configuration validation, numpy for metric aggregation. Core files include cli.py (entry point), bench.py (load engine), client.py (HTTP client), etc.

how-fast is suitable for scenarios such as gateway selection evaluation, configuration optimization verification, capacity planning, CI/CD performance regression testing, and SLO compliance proof. It is not a general HTTP stress testing tool but a precision instrument for LLM inference scenarios, helping optimize AI infrastructure through precise performance visibility.