Beyond FLOPs: A Practical Study on Pruning and Inference Acceleration of Large Language Models Based on GEMM Taxonomy

This study, published by the EIT-NLP team on arXiv in June 2026, systematically evaluates the real-world hardware acceleration effects of LLM pruning methods for the first time using a GEMM-centric taxonomy. It reveals the complex relationship between theoretical FLOPs reduction and actual inference speed. Key contributions include: proposing a GEMM taxonomy to unify pruning strategy evaluation, developing the PruningInferSim benchmark framework, discovering the Pareto optimality of static depth pruning and the phased transition of strategies with quality loss, and providing key guidance for model compression practices.

The expansion of LLM scales (from billions to trillions of parameters) has made inference latency and cost bottlenecks for deployment, and pruning is a mainstream optimization technique. However, there are misconceptions in the field: equating theoretical FLOPs reduction with actual acceleration, ignoring the complexity of hardware execution. Different pruning strategies (dynamic sparsification, depth pruning, attention head pruning, etc.) have significantly different impacts on underlying GEMM operations, requiring a unified evaluation framework.

GEMM-Centric Taxonomy: Reduces LLM inference to GEMM operations, mapping pruning strategies to three dimensions of GEMM: M (number of output rows/batch size), N (number of output columns/model width), and K (multiply-accumulate dimension), enabling unified comparison of strategies. PruningInferSim Framework: Features include: 1. Implementation consistency (eliminating biases from different implementations); 2. Hardware awareness (simulating memory bandwidth and computing capacity constraints); 3. Pareto frontier analysis (characterizing the trade-off between acceleration and quality loss).

1. Static Depth Pruning: Maintains the strongest Pareto-optimal baseline in all scenarios; acceleration in memory-constrained scenarios is close to the theoretical upper limit (coarse-grained pruning directly changes GEMM patterns with minimal hardware efficiency loss).
2. Strategy Transition in Prefill Phase: Low quality loss (0%-4%) → static depth pruning is optimal; medium (5%-16%) → dynamic depth pruning; high quality loss (17%-26%) → static width pruning.
3. Gap Between Theoretical and Actual Acceleration: Reasons include memory bandwidth bottlenecks, irregular computation patterns (fine-grained sparsity reduces cache efficiency), and special kernel launch overhead.

Implications for Practitioners:

• Choose strategies based on the quality tolerance of the application;
• Evaluations should be conducted on target hardware instead of relying solely on FLOPs;
• Depth pruning remains the most robust baseline for most scenarios.

Future Directions:

• Develop dynamic systems that adaptively select optimal strategies;
• Design pruning algorithms optimized for hardware architectures;
• Explore the synergistic effects of pruning with quantization and speculative decoding.

This study, through the GEMM taxonomy and PruningInferSim framework, provides the first strict and unified benchmark for evaluating the actual acceleration of LLM pruning, breaking the misconception that "FLOPs equals acceleration". In today's era of widespread LLM deployment, this measurement-based evaluation method will become an important cornerstone for progress in the field.