Ollama Benchmark: A Terminal Tool for Performance Stress Testing of Local Large Models

Ollama Benchmark is a terminal benchmarking tool designed specifically for Ollama local large models, offering comprehensive performance evaluation capabilities including GPU memory analysis, generation speed diagnosis, and concurrent stress testing. It addresses the pain point of lacking systematic performance evaluation tools in local LLM deployment, helping users accurately assess the actual operational performance of models under limited hardware resources and providing quantitative basis for hardware selection, model matching, etc.

With the surging demand for local deployment of large language models (LLMs), more and more developers and enterprises are choosing to run models locally instead of relying on cloud APIs. As one of the most popular local LLM runtime frameworks currently, Ollama greatly simplifies the process of model downloading, configuration, and operation. However, local deployment faces a core challenge: how to accurately assess the actual operational performance of models under limited hardware resources? Indicators such as GPU memory capacity, model loading overhead, and concurrent request processing capability directly affect the availability and user experience of local LLMs. Without systematic performance evaluation tools, users can only explore the matching scheme between hardware and models through 'trial and error'. Ollama Benchmark is born to solve this pain point, providing a complete terminal-level diagnosis solution.

The core features of Ollama Benchmark include:

1. Hardware-level Memory Analysis: Directly queries NVIDIA driver interfaces to accurately measure memory usage changes during different model operation stages, understanding resource consumption patterns in weight loading, context caching, concurrent requests, etc.
2. 5-Stage Performance Profiling: Evaluates performance in stages including baseline state, weight loading, active querying, saturated context, and concurrent stress, simulating real load changes to identify bottlenecks.
3. Speed and Latency Diagnosis: Measures indicators such as Prefill speed, generation speed, wall-clock time consumption, and parallel slowdown ratio to assess response capability in production environments.
4. Automated Log Export: Generates timestamped text logs and saves them to the output/ directory, facilitating data analysis and long-term tracking.

The technical implementation highlights of Ollama Benchmark include:

1. Direct Hardware Interface Call: Chooses to directly call nvidia-smi instead of high-level abstractions to ensure the accuracy of memory data, providing a reliable basis for capacity planning.
2. Concurrent Stress Simulation: Supports simulating multi-user concurrent scenarios, observing the inflection point of the performance curve by gradually increasing the number of requests to determine the optimal concurrent configuration.
3. Modular Architecture: Written in Python, supports uv and pip dependency management, and virtual environment activation scripts cover Windows, Linux, and macOS to ensure cross-platform compatibility.

The practical application scenarios of Ollama Benchmark include:

1. Hardware Selection Decision: Before purchasing a GPU, test the performance of the target model on existing hardware to provide a quantitative basis for procurement.
2. Model Selection Comparison: Quickly compare resource consumption and inference speed of different models on the same hardware to find the balance between performance and resources.
3. Production Capacity Planning: Evaluate the user scale that a single server can carry through concurrent stress testing, and formulate expansion strategies and load balancing solutions.
4. Performance Regression Detection: Incorporate logs into the CI/CD process to monitor the impact of model version updates or system configuration changes on performance.

The deployment process of Ollama Benchmark is simple:

1. Clone the repository and enter the directory
2. Install dependencies using uv sync or pip
3. Activate the virtual environment
4. Run python benchmark.py to start the test The tool provides command-line help options; use the -h parameter to view detailed configuration options and test mode descriptions.

Ollama Benchmark fills the gap in performance observation tools in the local LLM ecosystem; it is not only a speed tester but also a system-level resource diagnosis solution. For developers or teams who take local AI deployment seriously, this tool should be included in the standard toolchain. In today's mature AI infrastructure, 'how fast it runs, how much it occupies, and how many concurrent requests it can handle' are key to engineering implementation, and Ollama Benchmark is a professional tool to answer these questions.