LLMBenchmark: A .NET 10-based LLM Evaluation and Benchmarking Framework

LLMBenchmark is an LLM evaluation framework maintained by guizama and released on GitHub (link: https://github.com/guizama/LLMBenchmark, release date: 2026-06-16). Built on .NET 10 Minimal API, it addresses the core challenge of model selection in the LLM ecosystem, offering features like multi-model parallel execution, token estimation, deterministic validation, LLM-as-a-Judge automatic evaluation, PostgreSQL persistent storage, and scenario-driven benchmarking, supporting systematic evaluation of generation and rewriting tasks.

With the rapid development of the LLM ecosystem, developers and enterprises face the challenge of model selection—different models vary significantly in generation quality, rewriting ability, response speed, cost, etc., and it’s hard to judge based solely on vendor claims. Therefore, a systematic and reproducible evaluation framework has become an essential need, and LLMBenchmark was developed precisely to address this.

Multi-model Parallel Execution

Support for connecting multiple LLM provider models simultaneously, conducting parallel evaluations under the same scenario to ensure horizontal comparability, making it easy to compare the performance of OpenAI, Anthropic, local models, etc.

Token Estimation and Cost Control

Real-time token consumption estimation helps predict usage costs for cost-sensitive scenarios, enabling optimal model selection based on performance and cost.

Deterministic Validator

Built-in mechanism to detect model output stability; comparing outputs from multiple calls to the same model provides support for reliability assessment.

LLM-as-a-Judge Automatic Evaluation

Introduce LLM as a judge to score the quality of generated content, reducing manual annotation costs while maintaining objectivity.

PostgreSQL Persistent Storage

All results are stored in PostgreSQL, supporting historical data preservation, trend analysis, and version comparison to track the effects of model iterations or prompt strategies.

Scenario-driven Benchmarking

Allow users to define business scenarios and evaluation dimensions for targeted evaluation of tasks like customer service dialogue, code rewriting, copywriting creation, etc.

The framework adopts the .NET 10 Minimal API architecture, aligning with modern cloud-native application design trends: lightweight, high-performance, and easy to deploy. Minimal API reduces traditional MVC boilerplate code, making core logic clearer; the maturity and performance of the .NET ecosystem provide a guarantee for the stable operation of the framework.

LLMBenchmark has significant value for different roles:

• AI application developers: Provides objective basis for model selection;
• Model service providers: A standardized platform to showcase model capabilities;
• Researchers: A reproducible experimental environment. It is an important support for the healthy development of the LLM ecosystem.