oh-my-knowledge: A Scientific Evaluation Framework for Knowledge Input of Large Language Models

oh-my-knowledge is an open-source framework focused on evaluating the knowledge input of Large Language Models (LLMs). It provides systematic evaluation methods for prompts, RAG corpora, skills, and agent workflows, with built-in tools for statistical rigor (e.g., Bootstrap, Krippendorff Alpha) and debiasing mechanisms. Its core philosophy is to fix the model and vary the input to accurately measure the causal impact of knowledge input on model performance.

LLM applications are evolving from simple conversational assistants to complex knowledge work systems. Traditional evaluation methods have limitations (e.g., focusing only on final outputs, lacking statistical rigor). oh-my-knowledge proposes a core philosophy: fix the model and vary the input to accurately measure the causal impact of different knowledge inputs on model performance.

Prompt Engineering Evaluation

Support batch evaluation of multiple prompt versions, identify key influencing elements through controlled variables and statistical tests, and scientifically optimize prompt design.

RAG Corpus Quality Analysis

Provide tools for relevance scoring, information density analysis, redundancy detection, etc., to quantify the impact of corpus sources on generation quality.

Skill and Tool Evaluation

Evaluate metrics such as skill invocation accuracy, tool selection rationality, and execution efficiency to identify weak points in skill design.

Agent Workflow Evaluation

Track the execution status of each step in the workflow, analyze failure modes, and measure the impact of different designs on task completion rates.

Bootstrap Confidence Intervals

Provide reliable confidence interval estimates to help understand result variability and statistical significance.

Krippendorff Alpha Consistency Test

Quantify the consistency among human annotators to ensure the reliability of evaluation data.

Length Debiasing Mechanism

Eliminate the interference of output length on evaluation results to ensure fairness in comparing different input configurations.

Saturation Curve Analysis

Determine whether the evaluation sample size is sufficient and terminate data collection reasonably.

oh-my-knowledge brings an engineering evaluation methodology to LLM application development. In scenarios such as prompt optimization, RAG system tuning, and agent design, it helps developers make data-driven decisions instead of subjective judgments, which is crucial for building reliable production-grade AI systems.