Co-Design of Algorithms and Hardware: An Empirical Study on Optimizing Large Language Model Inference on Consumer GPUs

This study focuses on algorithm-hardware co-design, systematically evaluating the impact of low-precision quantization (e.g., INT8, INT4, AWQ) and structured sparsity techniques on LLM inference performance. It conducts cross-model validation on mainstream GPUs like T4, L4, and A100, revealing the deep correlation between optimization techniques and hardware characteristics, and provides data support for LLM deployment.

LLM inference deployment faces resource challenges (e.g., Llama3.1 8B requires 16GB of VRAM in FP16 mode). Existing optimization techniques include low-precision quantization (compressing weights to reduce memory and computation requirements) and structured sparsity (pruning redundant weights). However, different GPUs vary in their support for these techniques, so a systematic evaluation of their performance on different hardware is necessary.

Evaluated Models: Llama3.1 8B as the main model, supplemented by Llama3.2 1B and Qwen1.5-1.8B for cross-model validation; Tested Hardware: T4 (Turing architecture), L4 (Ada Lovelace architecture), A100 (Ampere architecture); Optimization Techniques: Quantization (BitsAndBytes INT8/INT4, AWQ), Sparsity (2:4 structured pruning, MaskLLM sparse mask); Evaluation Metrics: Throughput, memory usage, power consumption, energy efficiency, perplexity.

1. Quantization Benefits Are Hardware-Dependent: INT8 improves throughput in memory bandwidth-constrained scenarios, with more significant gains on A100; INT4 shows diminishing marginal returns, and may even experience performance regression due to dequantization overhead;
2. Sparsity as a Double-Edged Sword: Simple structured pruning leads to quality degradation, while the MaskLLM method preserves more capabilities; A100 has good support for sparse tensor cores, but T4/L4 have limited support;
3. Pareto Frontier for Energy Efficiency Optimization: The highest throughput configuration is not necessarily the most energy-efficient; medium precision (e.g., INT8) has outstanding energy efficiency, which is more valuable for edge deployment.

1. Avoid One-Size-Fits-All: The same model requires different optimization strategies on different GPUs;
2. Quantization Quality-Efficiency Trade-off: A small additional compression may lead to disproportionate quality loss;
3. Consider Full-Stack Costs: Integrate factors such as memory usage, power consumption, and model quality;
4. Hardware Evolution Direction: Understand the impact of GPU architecture evolution on optimization effectiveness to inform hardware procurement decisions.

Limitations: Focused only on NVIDIA GPUs, not covering AMD GPUs or dedicated NPUs; experimental model scales are small (8B and below); Future Directions: Explore mixed-precision strategies, composite optimization solutions, and dynamic inference scenarios (adaptive computation precision).

Algorithm-hardware co-design is key to full-stack optimization. This study breaks the perception that 'quantization is always good' or 'sparsity is always fast', providing empirical support and operational guidelines for building efficient and cost-effective AI systems. As Jensen Huang stated, performance leaps come from full-stack joint optimization, not improvements in a single component.