Roofline Model Analysis: Why Doubling Computing Power Doesn't Necessarily Make AI Faster

This article deeply analyzes the Roofline performance model, reveals the key role of memory bandwidth bottlenecks in LLM inference, breaks the cognitive misunderstanding that "computing power equals speed", and provides practical optimization ideas and interactive calculation tools to help understand the matching logic between hardware and workloads.

Misconception About Computing Power

There is a common misconception in the AI infrastructure field: buying more powerful GPUs/TPUs does not necessarily linearly improve inference speed, and the root cause lies in whether data can be delivered to computing units in time.

Core Position of FLOPs

FLOPs (floating-point operations) are the cornerstone of AI computing. Modern chips have computing power at the level of TFLOPS/PFLOPS, but their theoretical peak performance can only be achieved with data support.

Three Components of TPU Architecture

1. MXU (Matrix Multiplication Unit): Designed with a systolic array, it efficiently handles large-scale matrix operations but has low efficiency in small-batch inference;
2. HBM (High Bandwidth Memory): Weights, activations, and KV caches need to be loaded here. The bandwidth of H100 is 3.35 TB/s, and that of TPU v4 is 1.2 TB/s;
3. ICI (Inter-Chip Interconnect): 3D torus topology provides high-speed inter-chip communication and bypasses PCIe bottlenecks.

Definition of Arithmetic Intensity

Arithmetic intensity = total FLOPs / total bytes moved from HBM, which determines whether a workload is compute-bound or memory-bound:

• High arithmetic intensity: High data reuse rate and excellent computing efficiency;
• Low arithmetic intensity: Large proportion of data movement, with bandwidth becoming the bottleneck.

Visualization of the Roofline Model

• X-axis: Arithmetic intensity (FLOPs/Byte); Y-axis: Actual performance (FLOPs/s);
• Two lines: Memory bandwidth diagonal (bottleneck in low-intensity area), peak computing power horizontal line (bottleneck in high-intensity area);
• Ridge point: The intersection of the two lines, which distinguishes the critical point between memory-bound and compute-bound (about 295 FLOPs/Byte for H100, about 229 for TPU v4).

1. Autoregressive decoding feature: Generating a single token requires reading all weights, but the computation amount is minimal, leading to extremely low arithmetic intensity;
2. KV cache pressure: Long context windows increase KV cache size, exacerbating bandwidth demand;
3. Batch size limitation: Increasing batch size can improve arithmetic intensity, but actual batch size is limited due to HBM capacity and latency constraints.

Improve Arithmetic Intensity

• Quantization: INT8/INT4 reduces memory usage and bandwidth demand;
• Increase batch size: Continuous batching/speculative decoding improves effective batch size;
• Operator fusion: Merge small operations to reduce read/write of intermediate results;
• Paged attention: Optimize KV cache management to reduce fragmentation and bandwidth waste.

Architectural Innovation

• MoE: Activate only part of the parameters to reduce weight loading;
• Model parallelism optimization: Intelligent tensor/pipeline parallelism to reduce communication overhead;
• Near-memory computing: Place computing units close to memory to reduce data movement distance.

Open-source projects provide practical tools:

• Interactive calculator: Input model parameters/batch size/sequence length to automatically calculate arithmetic intensity and determine the bottleneck area;
• Python workload analyzer: Analyze memory access patterns and compute density of actual inference;
• Visualization script: Generate Roofline charts to intuitively display performance bottlenecks.

The Roofline model is not only a performance tool but also a mindset: Blindly pursuing peak computing power may waste resources, and it is necessary to match workload characteristics. LLM inference is mostly a battle with memory bandwidth; recognizing this is essential to formulating effective optimization strategies and realizing the value of AI hardware. As the author said: "Knowing the area where the workload lies is the premise of hardware and architecture decisions."