Snapdragon 8 Gen 3 Cross-Backend LLM Inference Benchmark: Mobile AI Performance Evaluation

This test conducts cross-backend large language model (LLM) inference benchmark tests on the Snapdragon 8 Gen3 flagship mobile platform, comparing the performance of three inference backends: CPU, GPU, and NPU. Evaluation metrics include inference speed, latency, power consumption, and energy efficiency. The tests cover mainstream open-source models such as Llama-2 7B and Llama-3 8B. Key findings: NPU has significant advantages in energy efficiency; GPU has outstanding performance but high power consumption; CPU is highly versatile but does not excel in either performance or energy efficiency. This provides important references for mobile LLM deployment.

From 2023 to 2024, mobile chip AI computing power achieved a qualitative leap. Flagship platforms like Snapdragon 8 Gen3 integrate dedicated NPUs (Hexagon NPU claims a 98% increase in AI performance and a 40% increase in energy efficiency), turning mobile devices' ability to run LLMs with billions of parameters from "barely usable" to "smoothly usable". However, releasing hardware capabilities requires software stack support. The performance of the same model running on different backends can differ by several times, so choosing the optimal backend is key to deployment.

Test Models: Selected open-source models including Llama-2 7B, Llama-3 8B, Mistral7B, and Qwen series, using the Q4_K_M quantization format to balance accuracy and model size; Inference Backends: CPU (ARM NEON optimized, high versatility), GPU (Adreno750, OpenCL/Vulkan parallel computing), NPU (Hexagon, QNN SDK optimized, optimal energy efficiency); Evaluation Metrics: Performance (Prefill/Decode speed, Time To First Token (TTFT), end-to-end latency), efficiency (power consumption, energy efficiency in tokens per Joule (tokens/J), temperature), stability (performance degradation, thermal throttling recovery).

Backend Performance: CPU Prefill:15-25 tokens/s, Decode:3-5 tokens/s, power consumption:3-5W; GPU Prefill:40-60 tokens/s, Decode:8-12 tokens/s, power consumption:5-8W; NPU Prefill:30-50 tokens/s, Decode:10-15 tokens/s, power consumption:2-4W. Model Differences: Llama-2 7B NPU optimization is mature; Llama-3 8B performs well on GPU; Mistral7B has obvious advantages in long context; Qwen series has good Chinese support. Energy Efficiency: NPU's energy efficiency is 3-5 times that of CPU; GPU has high performance but low energy efficiency; continuous load thermal throttling affects energy efficiency.

Backend Selection: Prioritize NPU (excellent energy efficiency, requires model optimization); GPU as an alternative (for short-term intensive computing); CPU as a fallback (for prototype verification). Quantization Strategy: Q4_K_M is the balance point; NPU needs to refer to vendor-specific quantization formats. Context Management: 4K is the sweet spot; above 8K requires KV cache management; Mistral's sliding window has significant advantages. Thermal Management: Intermittent inference, temperature monitoring, user options for performance-temperature trade-offs.

Limitations: Only tested 7B-8B models, not covering 13B/1B; backend implementation quality affects results; no dynamic load/multi-task testing; limited to the Snapdragon 8 Gen3 platform. Future Directions: Expand models and backends; add dynamic scenario testing; compare with other platforms (Dimensity, Tensor G3); track the performance of new chips (Snapdragon 8 Gen4).

This test verifies: 1. Edge-side LLMs are now practical (7B models reach 10+ tokens/s with NPU acceleration); 2. NPU is key to mobile AI (significant energy efficiency advantages); 3. There is large room for software optimization (backend implementation differences affect performance); 4. Quantization is a must (unquantized models are not practical). It provides empirical data for mobile LLM deployment and guides technical selection and optimization strategies.