llm-benchmark: A Personal LLM Model Evaluation Framework Supporting Local and API Model Comparison

llm-benchmark is an open-source personal LLM evaluation suite that supports comparison between Ollama local models and API models like Anthropic Claude and OpenAI GPT. It covers multi-dimensional test tasks including programming, reasoning, knowledge Q&A, output format compliance, and speed performance. The project emphasizes personalized customization (custom datasets, scenarios, hardware environments) to help users solve LLM selection dilemmas and provides an extensible performance evaluation tool.

With the booming development of the large language model ecosystem, developers face the dilemma of choosing between local lightweight models and cloud-based commercial APIs. Created by developer Jarkendar and developed in Python, llm-benchmark is an open-source evaluation suite that focuses on providing personal users with a customizable and extensible LLM performance evaluation tool, supporting simultaneous testing of local Ollama models and commercial API services.

Personalized Evaluation

Unlike general leaderboards, it supports users to use their own datasets, customize scenarios, and test on local hardware to compare the performance of privately deployed models and API models, which is close to actual application scenarios.

Dual-Mode Support

• Ollama Local Mode: Integrates local Ollama services, supports model series like Llama, Qwen, Gemma, ensuring data privacy and offline evaluation.
• API Cloud Mode: Supports Anthropic Claude and OpenAI GPT series, enabling comparison between local and cloud models through a unified abstraction layer.

1. Programming Ability: Evaluate code correctness, style, readability, and best practices (e.g., Kotlin tasks);
2. Reasoning Ability: Test the ability to analyze and derive complex problems;
3. Knowledge Q&A: Verify professional knowledge reserves and factual accuracy;
4. Output Format Compliance: Evaluate the ability to follow structured outputs like JSON/XML;
5. Speed Performance: Measure inference latency for different tasks.

Modular Components

• runner module: base_runner abstract interface, ollama_runner local executor, api_runner cloud executor;
• evaluator module: Uses Claude Sonnet as the referee model for automatic scoring;
• tasks module: Classifies tasks into coding/output_format/speed categories;
• dashboard module: Visualizes evaluation results.

Configuration-Driven

Manages model lists (Ollama/API) and evaluation parameters through YAML files, supporting flexible customization.

• Model Selection: Standardized comparison of candidate models' performance on own data;
• Local Optimization: Identify the optimal model in resource-constrained environments;
• Cost Analysis: Compare cost-effectiveness between local deployment and API calls;
• Iteration Tracking: Repeat evaluations to track performance changes of model versions.

Current Limitations

Limited dataset size, visualization functions to be enhanced, insufficient documentation examples, early-stage community ecosystem.

Future Directions

Integrate security/multi-language dimensions, distributed evaluation acceleration, model A/B testing, web interface to lower the threshold, community-shared evaluation dataset repository.

llm-benchmark provides a lightweight and fully functional open-source solution, advocating that evaluation should be close to actual scenarios rather than abstract leaderboards. It is suitable for developers and researchers to deeply understand the performance of models in specific use cases, and with community contributions, it will become a practical tool for LLM evaluation for individuals and small teams.