Infernum: An Open-Source Benchmarking Tool for Local Ollama Models

Infernum is an open-source command-line benchmarking tool specifically designed for local Ollama models. It supports multi-model performance comparison, cross-hardware comparison, and structured JSON output for easy automation integration. It addresses the pain point of standardized performance evaluation in local LLM deployment, establishes a community-driven performance database, and helps developers optimize deployment strategies and model selection.

With the popularization of local LLM deployment, developers need a standardized way to evaluate model performance on specific hardware. Traditional benchmarking relies on cloud services or complex configurations. As a lightweight CLI tool designed specifically for the Ollama environment, Infernum's core value lies in its simplicity and practicality, as well as its community performance database, which facilitates transparent performance comparison.

Basic Benchmarking

Run standardized tests with one click: infernum run --models llama3:8b,mistral:7b to generate results, publish them to the community, and provide a report link.

Multi-dimensional Comparison

• Cross-hardware: View performance differences of the same model on different hardware;
• Cross-model: Compare performance of multiple models on fixed hardware;
• Fine-grained filtering: Filter results by GPU model, memory, etc.

Structured Output

Supports the --format json parameter to output JSON data, making it easy to integrate into CI/CD or automation tools.

Developed in Go language to ensure cross-platform compatibility and efficient execution; static compilation simplifies deployment. Configuration uses YAML format (default path ~/.config/infernum/config.yaml) and supports custom parameters. It distinguishes between local testing and community services—can run offline, or optionally integrate community-contributed data, balancing privacy and sharing.

Model Selection Decision

Test candidate models on target devices to obtain real performance data, replacing theoretical indicators;

Hardware Performance Verification

Compare performance of old and new devices on the same model to quantify upgrade benefits;

Continuous Performance Monitoring

Combine JSON output with scheduled tasks to integrate into monitoring systems and detect performance degradation.

Currently in the early development stage with complete functions; future plans include supporting Homebrew installation to lower the threshold for macOS users. Long-term value depends on user participation and data accumulation—the community database will provide users with more comprehensive references and promote efficiency optimization of local LLM deployment.