GenAI-Bench: A Fine-Grained Performance Evaluation Tool for Large Language Model Inference Services

GenAI-Bench is a fine-grained performance evaluation tool designed specifically for LLM inference service systems, supporting token-level performance analysis to help developers accurately evaluate and optimize model service performance. It addresses the problem that traditional coarse-grained evaluations (such as overall latency and throughput) struggle to identify system bottlenecks, focusing on key metrics like Time to First Token (TTFT) and Token Per Output Time (TPOT), providing deep insights for the optimization of LLM inference services.

With the widespread application of LLMs in production environments, inference service performance optimization has become a key challenge. Traditional evaluation methods only provide coarse-grained metrics, making it difficult to identify system bottlenecks. In LLM inference scenarios, token-level performance characteristics are crucial: Time to First Token (TTFT) affects user-perceived response speed, while Token Per Output Time (TPOT) determines output fluency. There is a trade-off between the two, requiring fine-grained tools for analysis and optimization.

GenAI-Bench is a comprehensive token-level performance evaluation benchmark tool for LLM inference services. Unlike traditional end-to-end testing, it focuses on the generation process of each token, revealing: first-token latency distribution, stability of subsequent token generation speed, performance changes under different loads, and the impact of batching strategies on latency, helping developers deeply understand system behavior characteristics.

GenAI-Bench has several key features:

1. Token-level Latency Analysis: Precisely measures the generation time of each token, distinguishing between first-token and subsequent-token latency, helping identify bottlenecks (e.g., model loading, KV cache strategy optimization);
2. Multi-dimensional Load Testing: Simulates real-world scenarios such as different numbers of concurrent users, input/output length distributions, burst traffic, and sustained loads;
3. Integration with Mainstream Inference Frameworks: Compatible with popular frameworks like vLLM, TensorRT-LLM, and SGLang, providing a unified evaluation standard.

The core design idea of GenAI-Bench is to refine the evaluation granularity to the token level, with implementation methods including:

1. Precise Timing Mechanism: Uses high-precision timers to capture the generation time point of each token;
2. Streaming Response Parsing: Real-time parsing of inference service streaming output, recording the arrival time of each token;
3. Statistical Analysis Engine: Performs statistics on time data to generate metrics such as latency distribution and percentiles;
4. Visualization Report: Displays results in chart form to intuitively understand performance characteristics (e.g., long-tail latency, jitter).

GenAI-Bench is suitable for multiple scenarios:

1. Service Selection Comparison: Compares different LLM inference frameworks under a unified standard, making results more referenceable;
2. Configuration Optimization: Adjusts parameters such as batch size, KV cache strategy, and scheduling algorithm through token-level latency data;
3. Capacity Planning: Understands performance under different loads to determine GPU resources and instance counts;
4. Regression Testing: Ensures no performance degradation after system upgrades or configuration changes.

GenAI-Bench is developed by the SGLang project team, reflecting the open-source community's emphasis on LLM inference optimization. It not only provides tools but also establishes evaluation standards and methodologies, promoting industry performance optimization and best practice sharing. Future evolution will include: supporting more dimensions (memory, GPU utilization), multimodal models, richer visualization analysis, and integration with CI/CD pipelines for automated performance monitoring.