LLM-Para: A Roofline Analysis Framework for LLM Inference on Heterogeneous Multi-Level Memory Architectures

LLM-Para is a multi-metric first-order Roofline analysis framework aimed at solving performance analysis problems for large language model (LLM) inference on heterogeneous multi-level memory architectures. It supports modern LLM architectures such as GQA, MoE, and MLA, covers 24 hardware platforms, and provides multi-objective design space exploration capabilities to help users perform trade-off analysis across dimensions like performance, energy consumption, total cost of ownership (TCO), and carbon footprint.

As LLM scales grow exponentially, inference performance and efficiency have become core bottlenecks for deployment. Traditional analysis methods struggle to capture the nuances of modern architectures like GQA, MoE, and MLA. Engineering teams face the dilemma of lacking systematic quantitative tools when selecting hardware and optimizing deployments—empirical trial-and-error costs are high, and existing tools mostly focus on single dimensions (e.g., FLOPs or bandwidth) without comprehensive consideration of energy consumption, TCO, and carbon footprint.

The core contributions of LLM-Para include: 1. Heterogeneous multi-level memory model: For the first time, it systematically models the impact of chip-level multi-level memory hierarchies (such as SRAM, DRAM, NAND Flash) on decoding throughput, which is crucial for inference analysis of edge devices, mobile NPUs, and in-memory computing architectures; 2. Multi-objective Design Space Exploration (DSE) engine: It scans 5 hardware parameter dimensions and generates Pareto-optimal configurations for four objectives (performance, energy consumption, TCO, CO₂ emissions), facilitating early trade-off analysis.

LLM-Para supports analysis of 13 core operators (including attention mechanism-related ones like FlashAttention, feed-forward network-related ones like SwiGLU, and new architectures like MLA), covers 19 mainstream models (LLaMA-3, Mistral, Qwen2, Mixtral, DeepSeek-V2/R1, Gemma, etc.), and supports flexible quantization configurations from 2-bit to 32-bit.

LLM-Para covers 24 hardware platforms (NVIDIA GPU, AMD GPU, Apple Silicon, Intel, mobile NPU, in-memory computing, etc.). Key insights include: 1. Universal memory bottleneck in the decoding phase (arithmetic intensity ≤1 FLOP/Byte when batch size=1); 2. Trade-offs of MoE (selective loading reduces weight transfer but routing layer has low memory efficiency); 3. MLA trades computation for memory (32x KV cache compression but attention FLOPs increase by 500x); 4. NAND Flash quantization optimization (INT4 quantization can achieve a 35x throughput improvement); 5. Near-memory computing sweet spot (under energy constraints, bandwidth of 500-2000GB/s and computing power of 5-20TFLOPS enable over 20 tokens/s).

LLM-Para provides practical interfaces: 1. Web interactive interface (https://llm-para.onrender.com): Supports real-time parameter adjustment, interactive Roofline charts, FLOPs/memory decomposition charts, and data export; 2. Python CLI and API: Allows programmatic batch analysis of model-hardware combinations and rapid customization of analysis scenarios.

The value of LLM-Para for different roles: Algorithm researchers can verify the theoretical benefits of new architectures; system engineers can quantify hardware cost-effectiveness and bottlenecks; edge AI developers can evaluate the impact of quantization strategies; hardware architects can conduct early design space exploration to find the Pareto frontier of performance, energy consumption, cost, and sustainability.

LLM-Para promotes the evolution of LLM inference analysis from experience-driven to quantification-driven. By systematically modeling multi-level memory hierarchies, covering the complete operator set of modern architectures, and providing multi-objective optimization capabilities, it offers an open and scalable analysis benchmark for the community. As models and deployment scenarios diversify, this fine-grained performance modeling will become an essential tool for efficient AI system design.