In-depth Practical Testing of LLM Inference and Distributed Training: From Roofline Analysis to Quantization Strategies

This research conducts in-depth practical testing on the Llama 3.1 8B model using A100-SXM4-80GB hardware, covering Roofline performance bottleneck analysis, comparison of seven quantization strategies, research on attention mechanism variants, and distributed training stack analysis. It provides reproducible empirical data and optimization guidance, aiming to fill the gap between theory and practical test data in the field of LLM inference and training.

The project's motivation stems from the current situation in the LLM field where there are many theoretical articles but few practical test data. The core objective is to conduct comprehensive performance analysis on a representative model (Llama 3.1 8B) using production-grade hardware (A100), including single-token decoding Roofline analysis, quantization configuration comparison, attention variant implementation, and distributed training research. Each subdirectory contains runnable code, analysis reports, and practical test data to support reproduction and expansion.

Research Methods

1. Bottleneck analysis: Derive the Roofline model for single-step decoding on A100, calculate arithmetic intensity, time decomposition, and memory proportion
2. Quantization strategy comparison: Test 7 configurations including BF16 baseline, BnB INT8/FP4/NF4/NF4+DQ, GPTQ 4-bit, and AWQ 4-bit
3. Attention mechanism: Compare three implementations (eager/sdpa/flash-attention-2) and analyze MHA/MQA/GQA/SWA variants
4. Distributed training (in progress): Research DP/TP/PP/SP parallel strategies, FSDP sharding, LoRA fine-tuning, etc.

Tech Stack

Use tools like PyTorch 2.6+, transformers, bitsandbytes, AutoGPTQ, etc., run on A100-SXM4-80GB (RunPod) with CUDA versions 12.4-12.8, and provide detailed environment configuration instructions.

Key Findings

1. Decoding phase is memory bandwidth-bound: Arithmetic intensity of 0.7 FLOPs/byte vs A100's ridge point of 156, 99.5% of time spent loading weights
2. 4-bit quantization quality is consistent: NF4/GPTQ/AWQ all have a perplexity of 6.31 on WikiText-2
3. FP4 quality degrades: BnB FP4 has a perplexity of 6.66, the only obvious drop
4. INT8 has the worst performance: BnB INT8 throughput is only 7 tok/s with minimal quality improvement
5. GPTQ/AWQ have disk advantages: Pre-quantized model disk usage is only 5.3GB (vs 14.96GB for BF16)

Quantization Comparison Data

Method	Memory (GB)	Throughput (tok/s)	Perplexity	Disk (GB)
BF16 Baseline	14.96	33.21	5.92	14.96
BnB INT8	8.63	7.03	6.00	14.96
BnB FP4	5.76	22.30	6.66	14.96
BnB NF4	5.76	22.22	6.31	14.96
BnB NF4+DQ	5.43	17.94	6.31	14.96
GPTQ 4-bit	5.44	19.14	6.31	5.34
AWQ 4-bit	5.33	13.97	6.31	5.33

Note: Tests are based on HuggingFace generate() with batch size 1, not production deployment stack performance.

Core Conclusions

1. Optimization priority in decoding phase: Memory access patterns > computation optimization
2. 4-bit quantization selection: Minimal quality difference; choose based on deployment constraints (disk/flexibility)
3. Attention mechanism: GQA balances KV cache size and expressive power

Industry Implications

• Quantization selection guide: Choose GPTQ/AWQ for tight disk space, BnB NF4 for flexibility, avoid INT8
• Performance optimization direction: Focus on memory access (e.g., Flash Attention, KV cache optimization)
• Model architecture reference: GQA is a reasonable choice balancing performance and expression

Outlook

Once the distributed training part is completed, it will become a complete performance analysis reference from training to inference

Recommendations

1. Deployment scenarios: Select quantization methods based on disk space and throughput requirements
2. Performance optimization: Prioritize optimizing memory access patterns and use production-grade kernels (e.g., Marlin) to improve GPTQ/AWQ performance
3. Reproduction verification: Use project code to verify conclusions in the same hardware environment and expand research directions