Hidden Error Awareness in Chain-of-Thought: Models Know They're Making Mistakes but Choose to Stay Silent

This study reveals the phenomenon of hidden error awareness in large language models during chain-of-thought reasoning: models can internally detect their own reasoning errors (hidden state prediction accuracy reaches 0.95 AUROC), but their externally expressed confidence is almost indistinguishable from that of correct reasoning. This signal is only diagnostic (can judge whether reasoning is correct) rather than causal (cannot correct errors through existing interventions), challenging the assumption that chain-of-thought reasoning reflects internal computations.

The basic assumption of Chain-of-Thought (CoT) prompting is that the generated reasoning process reflects the model's internal computations. However, this study finds that this assumption does not hold: large language models can internally detect reasoning errors but exhibit high confidence in incorrect reasoning when expressing externally, a phenomenon that raises new questions for AI interpretability research.

By training linear probes on the model's hidden states, the study found: 1. The AUROC for predicting the correctness of reasoning trajectories reaches 0.95; 2. Even at the first reasoning step, the accuracy reaches 0.79. This indicates that the model's internal representations contain information about reasoning quality, which exists early on.

The externally expressed confidence of the model can barely distinguish between correct and incorrect reasoning:

Reasoning Type	Verbalized Confidence (out of 5)
Correct Reasoning	4.87
Incorrect Reasoning	4.55

The gap between the two is only 0.32 points. The AUROC of text-surface-based classifiers is only 0.59, far lower than the hidden state probe's 0.95, indicating that this internal information is not present in the generated text.

The study validated this phenomenon across multiple model families (Qwen, Llama, Phi series), parameter scales from 1.5B to 72B, and reinforcement learning-trained models (e.g., DeepSeek-R1 with AUROC 0.852), and found that the phenomenon exists stably.

Four intervention methods—activation guidance, probe-guided Best-of-N, self-correction, and activation patching—were tested and all failed. Core conclusion: The hidden error awareness signal is diagnostic (can judge whether reasoning is correct) rather than causal (cannot change the reasoning result).

Theoretically, reasoning error representations are different from factual knowledge representations (the former are uneditable), delineating the boundaries of mechanistic interpretability; practically, they can be used for error detection, human-machine collaboration, training data screening, and model evaluation.

Future explorations can include: 1. The neural basis of hidden signals; 2. The impact of training dynamics on signals; 3. Cross-task generalization; 4. More complex intervention methods.