benchpress: An LLM Inference Benchmark Tool Built for Apple Silicon

benchpress is an open-source LLM inference benchmark framework designed specifically for Apple Silicon (M1/M2/M3 series). Its core features include simultaneous evaluation of speed and generation quality, with rigorous statistical methods ensuring result credibility. It fills the gap in existing tools for consumer-grade Apple hardware, supports multiple backends, and is suitable for scenarios such as model selection, backend optimization validation, community contributions, and academic research.

Existing LLM benchmark tools have limitations: MLPerf focuses on data center-grade hardware, llm-benchmark only measures speed, and lm-eval only focuses on quality. Apple Silicon users lack a tool that simultaneously evaluates speed and quality with statistical rigor, so benchpress was created to fill this gap.

Speed Metrics

• tokens/sec (including bootstrap 95% confidence interval): core generation speed metric
• TTFT (Time to First Token): measures interactive response latency
• End-to-end latency: complete request processing time

Quality Metrics

• Perplexity: based on the WikiText-2 dataset; lower values indicate more accurate text prediction
• Task accuracy: evaluated on standard benchmarks like MMLU, HellaSwag, TruthfulQA
• Comprehensive quality score: an easy-to-compare score integrating multiple metrics

Statistical Rigor

• Paired Wilcoxon/Mann-Whitney U test: verifies the significance of performance differences
• Holm-Bonferroni correction: controls the overall error rate of multiple comparisons
• Cohen's d effect size: quantifies the magnitude of differences
• Thermal throttling detection: identifies overheating effects via Mann-Kendall trend test

Multi-Backend Support

• MLX (recommended): Apple-optimized framework leveraging Unified Memory and Neural Engine
• Ollama: user-friendly local LLM tool
• HuggingFace Transformers + MPS: PyTorch Metal backend
• llama.cpp (Metal): high-performance C++ implementation supporting quantized models

1. Model Selection: quickly compare the actual performance of open-source models on local hardware
2. Backend Optimization Validation: objectively compare the effects of migration (e.g., Ollama to MLX) or quantization schemes
3. Community Contribution: submit results to form a public leaderboard and accumulate real hardware data
4. Academic Research: provide standardized evaluation methodology to improve research reproducibility

• Command-line interface: clean and intuitive, supporting table, JSON, and Markdown outputs
• Progress visualization: displays progress bars during testing to enhance user experience
• Thermal management: supports run intervals (cooldown) to reduce thermal throttling effects
• Result export: JSON/Markdown formats for easy integration into CI/CD or documentation

Limitations

Primarily optimized for Apple Silicon; experience on other platforms needs improvement

Future Plans

• Quantization scanning: compare speed-quality tradeoffs across Q2-Q8 quantization levels
• GitHub Pages leaderboard: automatically render an online ranking
• PyPI and Homebrew distribution: simplify installation process

benchpress represents a new standard for consumer-grade LLM benchmark tools, balancing speed and quality, statistical rigor and flexibility. For LLM developers on Apple Silicon, it is an essential tool; for the AI ecosystem, its rigorous and transparent community-driven model will contribute to healthy development.