Ollama Benchmark: Local LLM Performance Testing Terminal Tool

Original Author/Maintainer: ysfemreAlbyrk Source: GitHub Original Link: https://github.com/ysfemreAlbyrk/ollama-benchmark Release Time: 2026-06-02

This tool is designed for Ollama local large models, providing multi-dimensional performance diagnosis including GPU memory usage, KV cache size, generation speed, etc. It helps users understand model resource consumption, evaluate inference efficiency, test concurrency capabilities, and optimize deployment decisions.

With the rapid development of LLM technology, more developers and researchers deploy models locally. Ollama is a popular platform for running models like Llama, Mistral, Qwen.

However, a core challenge exists: how to accurately evaluate a model's actual performance on specific hardware? Without systematic tools, users often choose models based on intuition, leading to resource waste or poor experience.

Key Testing Dimensions

1. Model Disk Usage: Measures local storage of each model, useful for users with many models.
2. GPU VRAM Monitoring: Tracks real-time VRAM allocation during model loading and inference.
3. KV Cache Evaluation: Measures KV cache usage under different context lengths to adjust max context window.
4. Speed Testing: Includes prefill speed (input processing) and generation speed (tokens/second).
5. Concurrency Pressure Test: Simulates multi-user requests to evaluate system performance.

Ollama Benchmark has the following technical advantages:

• Lightweight & No Dependencies: Terminal-based, no GUI, suitable for server environments.
• Real-Time Monitoring: Dynamically shows test progress and resource changes.
• Standardized Testing: Uses unified prompts and parameters for comparable results.
• Detailed Reports: Outputs structured data for further analysis.

Common Use Cases

1. Hardware Selection: Test model performance on existing hardware to guide purchase decisions.
2. Model Version Comparison: Compare quantized versions (Q4/Q8/FP16) to balance precision and speed.
3. Production Tuning: Use concurrency tests to identify bottlenecks and optimize parameters like thread count.
4. Academic Research: Establish standardized test processes for reproducible results in papers.

Ollama Benchmark fills an important gap in the local LLM ecosystem:

1. Data-Driven Decisions: From intuitive feeling to quantitative comparisons (e.g., Model A is 1.5x faster than B).
2. Maximize Resource Utilization: Understand hardware limits to avoid over-configuration or waste.
3. Quick Problem Localization: Diagnose performance issues (model/config/hardware).
4. Community Data Accumulation: Unified testing may form a public performance database.

Ollama Benchmark provides professional diagnostic capabilities for Ollama users. Regular benchmarking should be part of standard operations for local LLM applications.

Future expectations: Support multi-modal model testing, power consumption monitoring, and other dimensions to enrich the local LLM toolchain.