Axiom Framework: Systematically Evaluating the Confidence Calibration Capability of Large Language Models

Axiom is an open-source evaluation framework aimed at systematically measuring the confidence calibration performance of open-source large language models (LLMs) across multiple task types, including reasoning, common sense judgment, binary decision-making, and factual accuracy, helping developers identify models' overconfidence issues. The framework supports a variety of mainstream open-source models and provides dual-run modes (Kaggle and local). Its evaluation results have important guiding significance for model selection, fine-tuning, and product design.

Large language models often exhibit high certainty when generating answers, but this confidence does not necessarily match actual correctness—this is known as 'calibration mismatch'. In practical deployment, calibration mismatch poses serious risks: when enterprises rely on LLMs for decision support, medical diagnosis assistance, or financial analysis, users may make critical decisions based on incorrect high-confidence answers. Therefore, evaluating the calibration capability of LLMs is a necessary step before model deployment.

Axiom was developed by toxicskulll, with the core objective of providing a comprehensive confidence calibration evaluation framework for open-source LLMs. It is not limited to single-dimensional testing; instead, it explores differences in models' calibration performance across different task types, focusing on four main task categories: mathematical reasoning (e.g., GSM8K dataset), common sense understanding (CommonSenseQA), binary decision-making (BoolQ), and factual truth verification (TruthfulQA). This multi-dimensional analysis can reveal which domains models are prone to overconfidence in, providing references for developers and deployers.

Axiom's technical pipeline consists of three core stages: 1. Dataset Preparation: Automatically download and format authoritative evaluation datasets to ensure standardization and reproducibility; 2. Model Evaluation: Support batch inference and extract confidence signals—its innovation lies in using semantic answer evaluation combined with sentence embedding technology to judge the semantic equivalence between the model's answer and the standard answer, rather than strict string matching; 3. Analysis and Visualization: Calculate metrics such as Expected Calibration Error (ECE), Maximum Calibration Error (MCE), and Brier score, and generate reliability charts and confidence distribution graphs to intuitively display calibration performance.

Axiom is compatible with a variety of mainstream open-source models, including Llama 3.2/3.1 series, Google Gemma series, Mistral7B, Qwen2/3.5 series, DeepSeek LLM, Phi-4 series, TinyLlama, and Zephyr, etc. For gated models (such as Llama and Gemma), the framework provides clear configuration guidelines. In terms of usage, it offers three Kaggle Notebooks (divided into data preparation, evaluation, and visualization stages) to avoid GPU time limits, and also supports local operation (Python scripts + virtual environment configuration) to meet flexibility needs.

Axiom's evaluation results have guiding value across multiple scenarios: during model selection, calibration metrics can be used to screen for accurate and 'self-aware' models; during fine-tuning, training strategies that improve calibration performance can be identified; at the product design level, human-computer interaction strategies can be designed based on model calibration characteristics (e.g., prompting secondary confirmation when confidence is low). As LLMs are increasingly applied in key fields, confidence calibration will become a must-have in engineering practice, and Axiom provides a systematic tool for this purpose.

With its rigorous evaluation design and wide model support, Axiom provides a valuable open-source tool for LLM confidence calibration research. Today, as model capabilities are advancing rapidly, we not only need powerful models but also 'honest' models that can accurately recognize their own capability boundaries. Axiom is an important step in this direction.