ERR-EVAL: Evaluating AI Models' Cognitive Reasoning and Uncertainty Management Capabilities

ERR-EVAL is a benchmark focused on evaluating the cognitive reasoning capabilities of AI models, concentrating on two key dimensions: ambiguity detection and uncertainty management. It aims to address the issue where current mainstream models are overconfident and struggle to recognize their own limitations, providing a standardized evaluation tool and reference for building more reliable AI systems.

Large language models perform excellently in tasks like text generation and code writing, but in critical scenarios, the question of whether they can recognize their own limitations when faced with ambiguous or out-of-knowledge-range problems has become increasingly prominent. Cognitive reasoning (the ability to know what one knows and what one doesn't) is a basic human cognitive ability, but it is not innate in AI models. Mainstream models often give confident answers to all questions, even if the question is flawed or outside their training scope. ERR-EVAL was designed specifically for the systematic evaluation of this ability.

Ambiguity Detection Test Set

Covers various ambiguity types from real scenarios: referential ambiguity (e.g., vague references), semantic ambiguity (e.g., polysemy of "bank"), information missing (e.g., complexity problems without specific algorithms), boundary ambiguity (e.g., standards for "large files"), and implicit assumptions (e.g., questions with wrong premises).

Uncertainty Quantification Test

Evaluates the model's ability to express uncertainty: calibration (matching degree between confidence and actual accuracy), rejection strategy (rejection rate when unable to answer), and confidence expression (natural language description of the degree and source of uncertainty).

Comprehensive Scoring System

Multi-dimensional metrics: ambiguity recognition rate, clarification request rate, correct rejection rate, calibration error, overconfidence index.

Comparative Benchmark

By evaluating mainstream models like GPT-4 and Claude, identify the impact of architecture/training methods, version iteration changes, and difficulty differences across specific ambiguity types.

Common Defects

• Overconfidence: Still gives deterministic answers to obvious ambiguity problems, with few active clarifications;
• Domain differences: Better at recognizing uncertainty in math/programming domains, but prone to overconfidence in open-ended history/subjective judgment tasks;
• RLHF side effects: More "useful" but less willing to express uncertainty.

Non-linear Relationship Between Scale and Capability

The relationship between model scale and cognitive reasoning ability is not simply linear: for some metrics, larger models perform better, but the overconfidence problem is sometimes more severe, and simply expanding scale cannot solve it.

• Model selection reference: In high-risk scenarios (medical, legal, etc.), cognitive reasoning ability is more important than accuracy;
• Training improvement guide: Fine-grained results help identify improvement directions (e.g., add corresponding data if referential ambiguity performance is poor);
• System security assessment: Regular testing to monitor the model's cognitive reasoning performance and detect degradation after updates;
• UI design guidance: Design interfaces based on model limitations (e.g., prompt users to supplement context, require self-checks).

Current Limitations

• Language coverage: Mainly focuses on English, with limited coverage of ambiguities in other languages;
• Cultural context: Does not fully capture culturally specific ambiguities;
• Dynamic updates: Needs frequent test set updates to adapt to model capability improvements.

Future Directions

• Multilingual expansion: Add Chinese, Arabic, etc.;
• Multimodal evaluation: Expand to image and audio scenarios;
• Real-time interaction evaluation: Identify and clarify ambiguities in multi-turn dialogues;
• Adversarial testing: Design adversarial examples to test robustness.

ERR-EVAL represents a shift in AI evaluation from capability measurement to reliability and safety assessment. Ensuring that AI honestly faces its limitations is key to building trustworthy AI. It provides researchers and practitioners with tools to understand model behavior and guide improvements, emphasizing that "knowing what one doesn't know" is a necessary condition for achieving true intelligence.