llm_perf: A First-Principles Analysis Framework for Large Language Model Inference Performance

llm_perf is a lightweight, first-principles-based LLM inference performance modeling framework. Its core goal is to predict latency, throughput, and memory usage before building or renting a hardware cluster. It supports comprehensive analysis of the decoding phase, prefill phase, end-to-end metrics, and separate prefill/decoding. By replacing empirical testing with mathematical modeling, it helps reduce trial-and-error costs and accelerate system optimization iterations.

Traditional LLM inference performance analysis relies on post-deployment testing, which is costly and slow to iterate. llm_perf fills the gap in the system design phase, enabling answers to key questions before code deployment: Can the model run on specific hardware? What is the impact of different parallelization strategies (TP/PP/EP/SP) on performance? What are the resource requirement differences between the prefill and decoding phases? Is separate deployment worth it?

The core of llm_perf is a five-stage analysis pipeline: 1. Memory model (calculates memory usage for weights, activations, and KV cache); 2. FLOPs model (considers prefill and decoding FLOPs for MHA/GQA/MoE); 3. Traffic model (calculates HBM traffic, providing input for Roofline analysis); 4. Communication model (calculates collective communication time for TP/EP/SP/PP); 5. Latency model (predicts latency based on Roofline and overlap awareness). Additionally, it supports decoding pipeline (batching, B* analysis), prefill pipeline (chunked prefill), end-to-end metric assembly, separate deployment modeling, and framework overhead handling.

Case studies using GPT-1.8T MoE @ FP4 on the GB200 NVL72 configuration yielded: 1. The optimal chunk size C* for chunked prefill is approximately 2048 tokens; 2. HBM bandwidth affects parallelization strategies (low bandwidth prefers wide TP, high bandwidth makes TP a pure overhead); 3. Framework overhead has a significant impact on highly interactive scenarios but does not change the optimal partitioning choice; 4. Separate prefill/decoding is not worth it for short contexts (2-32k tokens) and only benefits long contexts of 64k+ tokens.

llm_perf uses a pure functional design (no global state), has a typed specification database (organized via JSON files), supports HuggingFace adapters (converts from HF config to model specifications), provides the DRAM3D tool (derives HBM bandwidth), and extracts the frontier of effective configurations via optimal partitioning Pareto scans.

llm_perf is suitable for: 1. Hardware procurement decisions (evaluating the applicability of different GPU configurations); 2. Parallelization strategy optimization (balancing latency and throughput); 3. Service capacity planning (predicting concurrent requests and QPS); 4. Architectural design trade-offs (evaluating benefits of features like separate deployment); 5. Performance bottleneck diagnosis (comparing predictions with actual measurements).

llm_perf represents a new paradigm for LLM inference performance analysis, shifting from empirical trial-and-error to first-principles modeling. Through rigorous mathematical modeling and rich case studies, it provides a powerful tool for LLM infrastructure planning and optimization, making it a valuable open-source resource for LLM service providers, cloud vendor AI teams, and researchers.