BenchForge: A Local LLM Performance Benchmarking Workbench

BenchForge is a local-first LLM benchmarking tool built on llama-bench. It supports automated performance testing of GGUF-format models in both CPU and GPU environments and provides an interactive comparison dashboard.

Local deployment of large language models (LLMs) has become the preferred choice for many developers and enterprises, as it protects data privacy and avoids the ongoing costs of API calls. However, local deployment faces a key challenge: how to accurately evaluate the performance of different models on actual hardware? The GGUF format (popularized by the llama.cpp project) allows quantized models to run efficiently on consumer-grade hardware, but the actual throughput and latency performance of different quantization levels and model architectures vary greatly across hardware configurations. BenchForge is designed to address this evaluation challenge.

BenchForge is a local-first LLM benchmarking workbench with an architecture combining a C++ core and a lightweight web frontend. Built on the mature llama-bench tool, it provides standardized performance testing and visual comparison capabilities for GGUF-format models.

BenchForge can automatically run a series of standardized tests to measure key performance metrics of models on specific hardware:

• Inference Latency: End-to-end response time for a single request
• Throughput: Number of tokens processed per unit time
• Perplexity Evaluation: Using standard datasets to measure the model's predictive ability
• Multi-configuration Testing: Supports comparative testing under different thread counts, batch sizes, and context lengths

The framework supports both pure CPU inference and CUDA/Metal-accelerated GPU inference testing, helping users understand the performance characteristics of models under different computing backends and providing data support for hardware selection.

After testing is completed, BenchForge launches a local web service (default port 7860) and provides an intuitive visual interface:

• Horizontal comparison charts of model performance
• Efficiency curves for different quantization levels
• Analysis of the relationship between hardware configuration and performance
• Trend tracking of historical test results

BenchForge uses a layered architecture design that balances performance and ease of use:

• Benchmark Module: Encapsulates llama-bench calling logic, manages test execution and metric collection
• Metrics Module: Standardizes calculation and storage of performance metrics
• Perplexity Module: Implements core algorithms for perplexity evaluation
• Discovery Module: Automatically scans and identifies local GGUF model files
• DB Module: Persists test results based on SQLite
• Server Module: Embeds an HTTP service to provide API interfaces for the frontend