llm-bench: MLX vs GGUF Inference Benchmark Framework for Apple Silicon

llm-bench is a comprehensive benchmarking tool designed specifically for Apple Silicon, aiming to systematically compare the inference performance of MLX (Apple's native framework) and GGUF (cross-platform format via llama.cpp) model formats. It covers multi-dimensional metrics such as prompt processing speed, generation speed, memory usage, and output quality, helping developers make data-driven technical choices. It is one of the signs of the maturity of Apple Silicon's local AI ecosystem.

With the rise of Apple Silicon (especially M-series chips) in the field of local LLM inference, developers face the technical dilemma of choosing between MLX and GGUF. The llm-bench project developed by haxlys is not just a simple speed testing tool but a systematic evaluation framework, aiming to isolate runtime differences and accurately measure performance gaps of the same model in different formats.

llm-bench evaluates performance from four key dimensions:

1. Prompt Processing Speed (PP)：Measures input prompt throughput (tokens/second), which is crucial for scenarios like long document understanding and RAG;
2. Token Generation Speed (TG)：Measures the speed of new token generation, directly affecting interactive chat experiences;
3. Memory Usage：Validates peak memory through dual verification using /usr/bin/time and MLX's mx.metal.get_peak_memory(), determining the maximum model size a device can load;
4. Output Quality：Calculates cosine similarity using sentence-transformers to evaluate semantic differences (often ignored by traditional benchmarks but critical for production).

Model Management：Driven by a YAML registry, preconfigured with Gemma4 26B-MoE (6 variants) and 31B Dense (2 variants); adding new models only requires modifying registry.yaml and running sync_models.py; Scenario Design：Prompt lengths (256/1024/4096/8192 tokens), generation lengths (128/512), and repetition times (3 formal runs +1 warm-up) simulate real-world usage; Toolchain：Supports model synchronization (auto-download missing variants), smoke testing (quick validation), full matrix testing, Streamlit visualization dashboard, and Quarto static report generation.

Preliminary test results based on Gemma4 26B-MoE:

• Speed Comparison：MLX-8bit has higher throughput in the prompt processing phase (thanks to Unified Memory and Metal optimization), but the gap may narrow or reverse when generating long sequences;
• Memory Efficiency：MLX 8-bit quantization uses slightly less peak memory than GGUF Q8_0, which is beneficial for loading larger batches or longer contexts;
• Output Consistency：Due to differences in quantization algorithms (MLX custom 8-bit vs GGUF Q8_0), outputs may have subtle semantic differences, requiring tool-based quantitative evaluation.

The project documentation emphasizes:

1. Avoid Metal Resource Competition：Close other MLX services (e.g., llm-stack) before running, otherwise performance may drop by 2-5x or cause OOM;
2. 重视 Preheating：Metal GPU needs preheating to reach a stable state; the tool has designed warm-up runs to eliminate variations;
3. Ensure Reproducibility：Record system version, MLX version, and llama.cpp version, and test in a controlled environment.

Ecosystem Significance：llm-bench helps developers make data-driven choices, demonstrate quantization benefits, track version evolution, and identify optimization opportunities, providing an empirical foundation for production LLM inference on Apple Silicon; Future Directions：Support more quantization schemes (MLX 4-bit, GGUF Q5_K_M, etc.), integrate more quality metrics (perplexity, downstream accuracy), batch concurrent testing, power consumption monitoring, etc., to solidify its position as a standard tool.