How Large Language Models Learn to 'Know What They Know and Admit What They Don't': Trace Inversion Enables AI to Proactively Say 'I Don't Know'

Researchers propose the Query Misalignment framework and Trace Inversion method, which detect the phenomenon of 'answering irrelevant questions' by analyzing model reasoning traces. This helps reasoning-focused large language models proactively refuse answering when uncertain, significantly improving their abstention ability across nine QA datasets. This method redefines the essence of hallucinations and provides a new defense line for AI safety.

Large language models (e.g., DeepSeek-R1, OpenAI o1) demonstrate strong reasoning abilities through Chain-of-Thought, but they have the hidden risk of 'overconfidence'—a lack of abstention ability: when faced with questions beyond their knowledge scope or with insufficient information, they do not refuse to answer but instead fabricate answers. In high-risk scenarios like healthcare and law, wrong answers have serious consequences, so saying 'I don't know' is more responsible.

The traditional view holds that hallucinations are wrong answers, but the authors propose a new perspective: many hallucinations are the model answering the 'wrong question'. Based on this, they put forward the Query Misalignment framework: when the model's internal reasoning process is misaligned with the user's original question, unreliable answers are generated, providing a new theoretical basis for error detection.

Trace Inversion is a three-step method based on the Query Misalignment framework:

1. Generate reasoning traces: Let the model produce a complete Chain-of-Thought process;
2. Reconstruct the query: Use an LLM to analyze the reasoning traces and restore the 'actual question the model answered';
3. Similarity comparison: Compare the semantic similarity between the original query and the reconstructed query to decide whether to trigger the abstention mechanism.

The study evaluated Trace Inversion on 4 large models (e.g., GPT-4, Claude) and 9 QA datasets:

• Outperformed baseline methods in 33 out of 36 experimental settings;
• Achieved stable improvements in fields like mathematical reasoning and commonsense QA;
• Zero-shot, no fine-tuning required. Compared to traditional methods, it directly detects the alignment between 'question and reasoning' and captures the dangerous situation of 'confident but wrong' answers.

The significance of Trace Inversion:

• Theory: Redefines hallucinations as misalignment between reasoning and user intent, opening up new research directions;
• Practice: Plug-and-play, no retraining or large-scale annotation needed;
• Safety: Serves as an additional defense line in high-risk scenarios, identifying reasoning deviations and refusing to respond.

Limitations:

• Requires generating detailed reasoning traces, increasing time and computational costs;
• The quality of reconstructed queries depends on the capability of the model used;
• The 'correct question' itself is ambiguous in vague questions. Future directions: Lightweight trace analysis, optimizing abstention strategies with reinforcement learning, and application in multimodal scenarios.

Trace Inversion reminds us: The reliability of large models lies not only in their knowledge reserve but also in their ability to recognize when reasoning goes off track. In an era of rapid AI capability advancement, teaching models to 'know what they know and admit what they don't' is a crucial step toward making them truly trustworthy.