LLM Inference Infrastructure Engineering Handbook: Building High-Performance Generative AI Systems from First Principles

This article introduces an open-source LLM Inference Infrastructure Engineering Handbook for AI infrastructure engineers. It provides physics-based interactive calculation tools covering key metrics such as throughput, latency, memory usage, GPU selection, and cloud cost modeling. It addresses resource waste and performance issues caused by reliance on vendor benchmarks or empirical configurations, helping to build efficient generative AI systems.

Currently, there are three major issues in LLM inference infrastructure decisions: over-reliance on vendor benchmark data under ideal conditions, random configurations without systematic methodology, and trial-and-error optimization. These issues lead to cost waste from over-provisioned GPUs, latency even at high costs, OOM crashes in large-scale deployments, and misunderstandings of system bottlenecks. LLM performance is determined by physical laws, not guesswork.

The handbook provides calculation tools across five dimensions:

1. Throughput Modeling: Differentiate between the compute-intensive Prefill phase (dependent on GPU computing power) and the memory-intensive Decode phase (dependent on memory bandwidth), and visualize bottlenecks;
2. Latency Prediction: Compute TTFT (Time to First Token), ITL (Inter-Token Latency), and system throughput to support latency-throughput trade-off analysis;
3. Memory Calculation: Cover model weights (parameter count + precision) and KV cache (proportional to batch size, sequence length, and number of layers; it can easily exceed the weight size in long-context scenarios;
4. GPU Selection: Balance single-card/multi-card deployment, Tensor Parallelism scaling, and constraints of interconnection bandwidth (PCIe vs NVLink);
5. Cloud Cost Modeling: Estimate monthly GPU costs, compare cloud vendor prices, and analyze cost impacts of auto-scaling and cold-start I/O.

Using the handbook provides five core insights:

1. The Decode phase is memory-bound rather than compute-bound; improving memory bandwidth is more critical;
2. In long-context scenarios, KV cache may exceed the model weight size; memory planning needs to be prioritized;
3. Batch size is a regulator for throughput and latency, requiring fine-tuning based on specific scenarios;
4. Multi-GPU scaling involves communication overhead (AllReduce), and interconnection bandwidth affects gains in concurrent scenarios;
5. Bandwidth is more important than TFLOPS in the inference phase (different from the training scenario).

The target audience includes AI infrastructure engineers, backend engineers transitioning to GenAI, ML engineers who need to deploy models to production, and platform teams operating frameworks like vLLM/TRT-LLM. Typical use cases: capacity planning, cost estimation, performance bottleneck diagnosis, GPU selection decisions, and team technical sharing.

The current version provides first-order estimates based on assumptions of standard Transformer architecture, optimized inference engines (vLLM/TRT-LLM), and dense models. Actual performance is affected by factors such as kernel efficiency and schedulers. Future plans include support for multi-node modeling, speculative decoding, real trace injection, auto-scaling simulation, and VLM memory modeling.

The field of LLM inference engineering is moving away from the era of empirical decision-making. The Infrastructure Engineering Handbook provides a systematic approach based on physical principles, helping teams build high-performance, low-cost generative AI systems. It is a practical tool for teams deploying large models in production environments.