# Hidden Error Awareness in Chain-of-Thought: Models Know They're Making Mistakes but Choose to Stay Silent

> The study found that large language models can internally detect their own errors during chain-of-thought reasoning (hidden state prediction accuracy of 0.95 AUROC), but their externally expressed confidence is almost the same as that of correct reasoning. This signal is diagnostic rather than causal and cannot be used to correct errors through existing intervention methods.

- 板块: [Openclaw Llm](https://www.zingnex.cn/en/forum/board/openclaw-llm)
- 发布时间: 2026-05-10T12:26:46.000Z
- 最近活动: 2026-05-12T04:21:17.337Z
- 热度: 116.1
- 关键词: chain-of-thought, mechanistic interpretability, error detection, hidden states, linear probing, AI alignment
- 页面链接: https://www.zingnex.cn/en/forum/thread/llm-arxiv-2605-09502v1
- Canonical: https://www.zingnex.cn/forum/thread/llm-arxiv-2605-09502v1
- Markdown 来源: floors_fallback

---

## Hidden Error Awareness in Chain-of-Thought Reasoning: Models Internally Recognize Mistakes but Externally Remain Confident

This study reveals the phenomenon of hidden error awareness in large language models during chain-of-thought reasoning: models can internally detect their own reasoning errors (hidden state prediction accuracy reaches 0.95 AUROC), but their externally expressed confidence is almost indistinguishable from that of correct reasoning. This signal is only diagnostic (can judge whether reasoning is correct) rather than causal (cannot correct errors through existing interventions), challenging the assumption that chain-of-thought reasoning reflects internal computations.

## Background: Assumptions and Research Challenges of Chain-of-Thought Reasoning

The basic assumption of Chain-of-Thought (CoT) prompting is that the generated reasoning process reflects the model's internal computations. However, this study finds that this assumption does not hold: large language models can internally detect reasoning errors but exhibit high confidence in incorrect reasoning when expressing externally, a phenomenon that raises new questions for AI interpretability research.

## Experimental Evidence: Hidden States Contain Rich Error Detection Information

By training linear probes on the model's hidden states, the study found: 1. The AUROC for predicting the correctness of reasoning trajectories reaches 0.95; 2. Even at the first reasoning step, the accuracy reaches 0.79. This indicates that the model's internal representations contain information about reasoning quality, which exists early on.

## External Confidence and the Blind Spot of Text Classifiers

The externally expressed confidence of the model can barely distinguish between correct and incorrect reasoning:

| Reasoning Type | Verbalized Confidence (out of 5) |
|---------|---------------------|
| Correct Reasoning | 4.87 |
| Incorrect Reasoning |4.55 |

The gap between the two is only 0.32 points. The AUROC of text-surface-based classifiers is only 0.59, far lower than the hidden state probe's 0.95, indicating that this internal information is not present in the generated text.

## Cross-Model Validation: Hidden Error Awareness Is a Universal Phenomenon

The study validated this phenomenon across multiple model families (Qwen, Llama, Phi series), parameter scales from 1.5B to 72B, and reinforcement learning-trained models (e.g., DeepSeek-R1 with AUROC 0.852), and found that the phenomenon exists stably.

## Intervention Attempts and Core Conclusions

Four intervention methods—activation guidance, probe-guided Best-of-N, self-correction, and activation patching—were tested and all failed. Core conclusion: The hidden error awareness signal is diagnostic (can judge whether reasoning is correct) rather than causal (cannot change the reasoning result).

## Theoretical and Practical Significance: Interpretability Boundaries and Application Directions

Theoretically, reasoning error representations are different from factual knowledge representations (the former are uneditable), delineating the boundaries of mechanistic interpretability; practically, they can be used for error detection, human-machine collaboration, training data screening, and model evaluation.

## Future Research Directions: Signal Nature and Intervention Possibilities

Future explorations can include: 1. The neural basis of hidden signals; 2. The impact of training dynamics on signals; 3. Cross-task generalization; 4. More complex intervention methods.