llm-bench: Panoramic Evaluation of Cross-Platform Large Model Inference Performance, 5100+ Real-World Data Reveal Hardware and Engine Differences

The llm-bench project, through systematic cross-platform evaluation, provides performance data for the Qwen3.5 series models covering 4 hardware platforms, 5 inference engines, and over 5100 measurements. It aims to serve as a data-driven reference benchmark for local large model deployment and help solve key issues in selecting inference engines for specific hardware.

In recent years, local large model deployment has evolved from a geek toy to a production option, but it faces the complexity of explosive hardware and software combinations. Hardware diversity includes Apple Silicon (unified memory architecture), NVIDIA GPU (mature CUDA ecosystem), AMD processors (Ryzen AI with integrated NPU), and multi-card configurations (expanding VRAM but with communication overhead). The inference engine ecosystem covers llama.cpp (cross-platform, quantization support), vLLM (high-throughput optimization), TensorRT-LLM (official NVIDIA optimization), MLX (deeply optimized for Apple Silicon), and Ollama (user-friendly encapsulation).

The llm-bench evaluation covers three core dimensions: hardware platforms (Apple Silicon, NVIDIA DGX Spark, AMD Ryzen AI MAX395, RTX3090×2), inference engines (5 mainstream engines), and model sizes (Qwen3.5 series from 9B to 122B). With over 5100 measurements, it ensures statistical significance and result reliability, revealing performance distributions, edge cases, and cross-configuration patterns for different setups.

The evaluation reveals the importance of hardware-engine matching (no universal optimal configuration—e.g., Apple Silicon may perform best on MLX, while NVIDIA hardware may excel on TensorRT-LLM/vLLM); non-linear scaling of model sizes (performance degradation is non-linear, affected by memory bandwidth, quantization strategies, and memory management efficiency); and the trade-off between quantization and precision (performance of different quantization levels is crucial for resource-constrained scenarios).

llm-bench provides developers with multi-faceted references: hardware selection (best cost-effectiveness within budget, whether high-end hardware is needed for specific model sizes, value of multi-card configurations); engine selection (benefits of switching engines for existing hardware, optimization for low-latency/high-throughput setups); and model size decisions (whether small models are sufficient, trade-off between resource consumption and benefits of large models).

llm-bench embodies the value of scientific evaluation: reproducibility (public code and experimental setups), standardized metrics (unified tokens/second for cross-platform comparison), and continuous updates (maintaining timeliness with iterations of new hardware/engines).

Current limitations include a single model family (only Qwen3.5), specific workloads (fixed prompt/generation lengths), and software version sensitivity. Future expansion directions: incorporating more model architectures, testing long-context performance, evaluating multimodal capabilities, adding power consumption metrics, and testing concurrent stability.

Through large-scale systematic evaluation, llm-bench provides a valuable data foundation for local LLM deployment, and its real-world measurement guidelines are more practical than theoretical analyses. We look forward to more similar evaluations to promote transparency and maturity in the local AI deployment ecosystem.