Can Large Language Models Truly Recognize Their Own Errors? A Cross-Format Transfer Study on Error Awareness Detection

This article focuses on the key question: "Can large language models recognize their own errors?" Researchers developed a low-cost error awareness detector based on probability distributions, but cross-format transfer tests revealed that the detector does not truly understand errors—instead, it overfits to surface features of the dataset. This finding has important implications for the reliability assessment of LLMs and AI safety.

As LLMs are applied in high-risk scenarios like medical diagnosis and legal consultation, their error awareness ability (whether they can recognize errors in their own outputs) has become key to improving reliability. The recently proposed probability distribution detection method is low-cost (single forward pass) and achieves an AUC of 0.88-0.99 in specific benchmark tests, but whether this success reflects the model's intrinsic ability is questionable.

The study uses the "commit-probability probe" method: prompt the model to end sentences with a period, then read the P(".") probability as the error awareness signal. While in-distribution tests performed well, performance dropped sharply in cross-format transfer tests, indicating that the detector did not learn model-level error awareness mechanisms—only fitting surface features of specific datasets.

Ironically, two simple baseline methods outperformed the complex detector across all cross-format tests: 1) P(?) baseline: read P("?") + P(" ?") probabilities as the error score; 2) P(True) baseline (Kadavath 2022): rephrase sentences into true/false judgments and calculate P(A)/(P(A)+P(B)). Experiments show these two methods outperformed the full detection pipeline in all cross-format test units.

The study constructed multiple datasets: arithmetic_dataset (50,000 arithmetic problems), capital_dataset (360 capital city questions), currency_dataset (216 currency questions), language_dataset (242 language questions), fever_dataset (180,000+ fact verification data), mmlu_math_dataset (2992 MMLU math problems), truthfulqa_dataset (1592 TruthfulQA questions), liars_bench_dataset (20,000+ deceptive dialogues). The models cover 11 open-source models from five families: Gemma, Llama, Mistral, Phi, and Qwen, with parameter sizes ranging from 2B to 27B.

Feature importance analysis reveals that the detector relies heavily on dataset-specific vocabulary and syntactic patterns rather than semantic content. For example, a detector trained on the arithmetic dataset overfocuses on number formats and operators—features that cannot generalize to other knowledge-based questions. This suggests that we cannot assert model capabilities based solely on excellent performance in specific benchmarks; strict out-of-distribution tests are needed for verification.

This study is published as a "failure report", highlighting the value of negative results. The team has made the code, data, and experimental procedures public to provide references for future research. In practice, deploying LLM monitoring tools based on probability distributions requires caution, as their reliability in complex real-world scenarios is questionable. Future directions include: developing format-agnostic error awareness methods, exploring the relationship between model internal representations and error awareness, and establishing stricter cross-domain evaluation benchmarks.

The error awareness ability of LLMs remains an open question. Through rigorous experiments and large-scale cross-model evaluations, this study reveals the limitations of current methods and provides a corrective signal for the field's development. On the path to more reliable and trustworthy AI systems, such critical research is indispensable.