Comprehensive Evaluation of Apple Silicon LLM Inference Performance: 8 Backends, 7 Models, 791 Sets of Actual Test Data

This article introduces the apple-silicon-llm-bench project, which conducts systematic benchmarking of LLM inference performance on the Apple Silicon platform. It covers 8 major inference backends, 7 mainstream models, and collects a total of 791 sets of actual test data, aiming to provide objective data support for Mac users to choose local LLM solutions.

apple-silicon-llm-bench is a standardized benchmarking project specifically for the Apple Silicon platform. Unlike scattered tests, it uses a unified method to evaluate mainstream backends and models. Its core objective is to eliminate information asymmetry, provide reproducible performance data, and help users choose appropriate local LLM solutions.

The tests cover 8 inference backends (e.g., llama.cpp, MLX, TensorFlow Lite, etc.) and 7 mainstream models (including Llama 2, Mistral, Qwen, etc., with parameter sizes ranging from 7B to 70B), accumulating 791 sets of data. Test metrics include tokens/second, memory usage, and first response time. All tests are conducted in a controlled environment to ensure comparability.

1. Different inference backends show significant performance differences on Apple Silicon; some backends have throughput several times higher than others on specific models. 2. Memory bandwidth is a performance bottleneck, and the unified memory architecture of Apple Silicon has obvious advantages. 3. Proper quantization can improve inference speed and reduce memory usage with almost no loss of quality, which is crucial for consumer-grade Macs to run large-parameter models.

• General users: Answers the question of 'what models can a Mac run'; - Developers: Choose inference backends suitable for their scenarios; - Researchers: Optimize model deployment strategies. In addition, the unified memory design of Apple Silicon reduces data transfer overhead, which has prominent advantages in memory-intensive LLM inference.

Limitations: The tests focus on inference performance and do not cover training/fine-tuning scenarios; continuous updates are needed to keep up with the development of new models/backends. Future plans: Continuously update data, welcome community contributions of more backend and model test results to maintain the project's timeliness.