Apple Silicon Local LLM Inference Benchmark: Performance Comparison Between llama.cpp and MLX Engines

This project is maintained by Ruben-Alvarez-Dev, aiming to fill the gap in performance comparison between llama.cpp and MLX engines on Apple Silicon. Through standardized tests on Mac Mini M1 (entry-level) and MacBook Pro M1 Max (flagship-level), it provides developers with data-driven guidance on framework selection, hardware configuration, model choice, etc., to help run LLMs efficiently in local environments.

With the development of LLM technology, the demand for local deployment is growing (low latency, privacy protection, cost control). Apple Silicon's unified memory architecture and Neural Engine have become popular choices, but the community lacks systematic comparison data between llama.cpp (a popular cross-platform framework) and MLX (Apple's official framework). This project was thus created to provide a standardized benchmark suite.

Hardware: Mac Mini M1 (8-core CPU/8-core GPU, 8GB/16GB RAM) represents budget configurations; MacBook Pro M1 Max (10-core CPU/32-core GPU/16-core Neural Engine, up to 64GB RAM) represents flagship performance.

Frameworks: llama.cpp (cross-platform, multiple quantization schemes, Metal optimization, active community); MLX (Apple-native, unified memory utilization, deep Metal integration, Python-friendly API).

Performance is evaluated from 5 dimensions:

1. Inference latency (first token time, subsequent token speed);
2. Throughput (number of requests processed per unit time / number of tokens generated);
3. Memory usage (determines the size of models that can be run);
4. Impact of quantization (performance changes with different levels Q4_0/Q5_K_M/Q8_0);
5. Model compatibility (support and performance differences across architectures like Llama/Mistral/Qwen).

Although specific values need to be referenced from the original report, the following can be expected:

• MLX's native optimization may bring latency/energy efficiency advantages (especially in Neural Engine utilization);
• llama.cpp's quantization schemes are more mature and diverse, suitable for memory-constrained scenarios;
• M1 Max can run 70B-scale models, while M1 Mini is suitable for 7B-13B models;
• Different model architectures show varying performance in the two frameworks, so selection should be based on needs.

Practical Guidance: Helps with hardware purchase, framework selection, model size determination, quantization strategy choice, and performance tuning.

Reproducibility: Provides complete test scripts and configuration instructions, supports community verification and extended testing, has high transparency, and reflects performance in real user scenarios.

Limitations: Currently only covers two devices in the M1 series.

Future Directions: Expand to M2/M3 chips; add more model architecture tests; evaluate long-term running stability; test multimodal model inference performance.

This project provides valuable performance data for local LLM inference on Apple Silicon, allowing developers to make decisions based on metrics like latency, throughput, and memory usage instead of guesswork. It is an important reference for LLM development and deployment in the Apple ecosystem.