Kriterion: An Open-Source Large Language Model Evaluation Framework Using an Independent Judgment System to Scientifically Compare Model Capabilities

Kriterion is an open-source large language model evaluation framework based on an independent judgment mechanism, designed to address the problem of objectively comparing model capabilities amid the explosion of open-source LLMs. Through a multi-dimensional evaluation system and independent judgment models, it scientifically measures model performance across dimensions such as factuality, reasoning ability, instruction following, and format compliance.

Due to the generation of open-ended text, traditional LLM evaluation methods have limitations: benchmark tests struggle to reflect real-world scenarios; manual evaluation is costly and has poor reproducibility; automated metrics (e.g., BLEU, ROUGE) often do not align with human subjective perceptions. These issues have driven Kriterion to adopt an independent judgment model approach.

Multi-Dimensional Evaluation System

Covers four core dimensions:

• Factuality: Assess content accuracy, avoiding hallucinations and misinformation;
• Reasoning Ability: Test multi-step reasoning such as logic, mathematics, and causal analysis;
• Instruction Following: Measure the ability to understand and execute user instructions (format, content, style);
• Format Compliance: Check if outputs conform to structured formats (JSON, tables, etc.).

Independent Judgment Mechanism

Evaluate outputs using independent judgment models. Advantages include flexibility (adapting to new scenarios), semantic understanding (recognizing equivalent expressions), and scalability (adjusting prompts to iterate standards). Biases or limitations of the judgment model are mitigated through carefully designed prompts and multiple validations.

Test Set Construction

Uses a test set of 200 carefully designed prompts with the following features:

• Diversity: Covers tasks such as knowledge Q&A, creative writing, and code generation;
• Difficulty Gradient: Ranges from simple factual queries to complex reasoning;
• Practical Relevance: Prioritizes questions from real usage scenarios.

Model Comparison Experiments

Conducted comparative evaluations of three open-source weight models. Results are presented in a visual dashboard, intuitively showing each model's scores across dimensions and responses to specific cases, providing references for users to select models.

Applicable to multiple scenarios:

• Model Selection: Provides objective data for enterprises/developers to choose models suitable for their scenarios;
• Iteration Monitoring: Serves as a regression testing tool to ensure model versions do not degrade;
• Academic Research: Validates the effectiveness of new model architectures or training methods;
• Educational Demonstration: Helps learners understand the complexity of LLM evaluation.

Limitations

• Judgment Model Dependence: Evaluation quality is affected by the capabilities of the judgment model;
• Limited Evaluation Dimensions: Does not cover dimensions such as creativity, multilingualism, and safety;
• Test Set Scale: The 200 prompts need to be expanded to fully evaluate general-purpose LLMs.

Future Directions

Introduce cross-validation with multiple judgment models, expand evaluation dimensions, build larger test sets, and develop detailed scoring standards.

Kriterion provides a valuable tool for open-source LLM evaluation. In the field of rapid model iteration, a reliable evaluation system is crucial for driving technological progress and responsible application deployment. Through systematic multi-dimensional evaluation and an independent judgment mechanism, it helps developers clearly understand the characteristics of model capabilities, contributing to the healthy development of the AI ecosystem.