APORIA: A Rigorous Evaluation Framework for Metacognitive Capabilities of Large Language Models

APORIA is a rigorous evaluation framework for the metacognitive capabilities of large language models (LLMs). Its core lies in using a dynamic five-round interaction protocol to isolate interfering factors and assess metacognitive abilities such as self-reflection and confidence calibration. This framework fills the gap in the current neglect of the metacognitive dimension in LLM evaluations and is of great significance for improving model reliability and safety.

Metacognition refers to the cognition and regulation of one's own cognitive processes, which is a hallmark of advanced intelligence. Current LLM evaluations mostly focus on knowledge reserve and reasoning abilities, but ignore the model's awareness of its own cognitive state (such as the problem of "not knowing what one doesn't know"), which brings deployment risks. The name APORIA comes from the Greek word meaning "perplexity", reflecting the challenge of metacognitive evaluation, and aims to systematically assess the self-awareness capabilities of LLMs.

APORIA's dynamic five-round interaction protocol simulates real scenarios: 1. Initial question + confidence expression; 2. Introduce challenge/contradictory information; 3. Explain the reasoning process; 4. Set traps/misleading information; 5. Summarize and reflect. Through multi-round dialogues, we observe the model's metacognitive performance, such as its ability to respond to challenges, correct opinions, and resist misleading information.

To eliminate interference, APORIA adopts strict isolation principles: 1. Control knowledge interference (avoid relying on domain-specific knowledge); 2. Reduce language comprehension interference (clear expression + clarification mechanism); 3. Consider context window limitations (adapt to the capabilities of mainstream models), ensuring that the evaluation focuses on metacognition itself.

APORIA evaluates from four dimensions: 1. Confidence calibration (matching confidence level with accuracy); 2. Error identification (real-time/post-hoc error correction); 3. Strategy adjustment (optimize reasoning based on feedback); 4. Awareness of knowledge boundaries (identify knowledge blind spots and express uncertainty).

Experiments show: 1. Significant differences in metacognitive performance among different model families; 2. Model size and metacognition are not simply positively correlated (some dimensions are less sensitive to size); 3. Task fine-tuning may lead to an "ability-metacognition trade-off" (improvement in target performance but decline in metacognition).

In applications, APORIA guides model selection (prioritize models with strong metacognition in high-reliability scenarios), improvement (targeted optimization of training strategies), and safety assessment (identify risks). Future directions include expanding multilingual evaluation, improving automation, adding emotional/social metacognitive dimensions, and involving the community to refine the framework.