Open-source LLM Inference Performance Test on Apple Silicon: A Comprehensive Evaluation of the MLX Framework

This article uses the LLM-Inference modular benchmark suite based on the MLX framework to systematically evaluate the impact of quantization strategies, KV cache optimization, and prefill technology on LLM inference performance on Apple Silicon devices. It provides developers with reproducible, systematic performance evaluation tools and data support to facilitate the optimized deployment of edge AI applications.

With the improvement of large language model (LLM) capabilities, developers hope to run models on local devices, but the performance of open-source models on consumer-grade hardware has uncertainties: How much precision is lost after quantization? How much speed improvement does KV cache optimization bring? How does memory usage change with different configurations? To address these issues, the open-source community has launched the LLM-Inference project, specifically designed for Apple Silicon, which builds a reproducible performance evaluation tool based on the MLX framework.

LLM-Inference adopts a highly modular architecture with the core concept of "composability", allowing developers to freely combine optimization strategies. It supports four weight quantization levels: fp16 (native bf16 baseline), 8-bit, 4-bit, and 2-bit; and provides two optimization switches: KV cache compression (reducing full precision to 4-bit) and prefill optimization (extending the step size from 512 tokens to 2048 tokens with tiling processing). A single evaluation covers 16 configuration combinations, providing complete data to understand the marginal benefits of quantization and optimization.

Weight Quantization Implementation

fp16 uses the native bf16 format as the baseline, while 8-bit/4-bit/2-bit are implemented via community quantization models. Support varies across models (e.g., Llama3-8B supports 2-bit, Mistral/Qwen require manual configuration).

KV Cache Compression Strategy

Compressing KV cache from full precision to 4-bit significantly reduces memory usage while maintaining reasonable precision. This is crucial for long-context scenarios, enabling 24GB devices to handle longer sequences.

Prefill Tiling Technology

Extending the prefill step size to 2048 tokens with tiling processing reduces GPU kernel launch overhead, improves large-scale batch processing throughput, and optimizes the Time To First Token (TTFT) for interactive applications.

Testing Llama3.1-8B and Mistral-7B on a 24GB memory M3 Mac:

• Memory-constrained scenarios: The w4+kv_cache configuration reduces memory usage by 60-70% compared to pure fp16, with controllable throughput loss;
• Extreme speed scenarios: Enabling prefill optimization can reduce the first token generation time for long contexts by 30-50%;
• Qwen32B encountered an OOM error due to its large parameter size, and the project automatically detected and skipped it, demonstrating robustness.

LLM-Inference establishes a "data-driven" model selection methodology, allowing developers to find the optimal balance between precision, speed, and memory through actual tests based on hardware configurations and scenarios. It fills the gap in open-source LLM performance benchmarking for the Apple Silicon ecosystem, and with the iteration of MLX and the open-sourcing of quantization models, it will become an important reference for edge AI development.

LLM-Inference demonstrates the open-source community's active exploration in edge AI optimization, providing practical tools through modular design and systematic testing. In the future, it is expected to expand support for more model architectures, optimization strategies, and cross-platform comparisons, providing a more comprehensive technical reference for edge large model deployment.