Do Large Model Explanations Make People More Gullible? Study Reveals the "False Trust" Trap

Studies have found that while LLM reasoning traces and post-hoc explanations increase user acceptance, they fail to help identify incorrect answers and instead create "false trust". Only comparative dual explanations can truly enhance users' ability to distinguish between correct and incorrect AI outputs. This article will discuss the background, experiments, results, and implications around this core viewpoint.

Problem Background

Large Language Models (LLMs) and Large Reasoning Models (LRMs) are being applied to critical tasks such as medical diagnosis and legal consultation, but cannot guarantee correct answers. To help users judge trustworthiness, the mainstream approach is to provide traceability explanations like reasoning traces, summaries, or post-hoc explanations. However, there is a fundamental tension: do these explanations help identify errors, or do they merely persuade users to accept answers regardless of correctness?

Experimental Design

The study used a between-group design to simulate real scenarios where users cannot independently verify AI answers. Four explanation conditions were tested:

1. Reasoning trace (complete step-by-step reasoning)
2. Reasoning summary (concise overview of the trace)
3. Post-hoc explanation (additional explanation after answer generation)
4. Comparative dual explanation (simultaneously presenting supporting and opposing arguments) The core metric is "false trust"—the degree of user trust in incorrect AI answers.

Key Results

• Trap of Reasoning Traces and Post-hoc Explanations: These explanations significantly increase user acceptance, but indiscriminately (regardless of answer correctness), creating cognitive bias—users believe the conclusion due to the "reasonable" reasoning process without evaluating its validity.
• Breakthrough of Comparative Dual Explanations: The only condition that improves users' discrimination ability, allowing them to better distinguish between correct and incorrect outputs.

Theoretical Significance

• Persuasiveness vs. Informativeness of Explanations: Traditionally, explanations are thought to enhance transparency, but this study shows they may only be persuasive—leading users to accept AI authority rather than make independent judgments, which is more dangerous in high-risk scenarios.
• Critical Thinking Requires Counterperspectives: Comparative explanations are effective because they force users to weigh different arguments, echoing mechanisms like legal debates and peer review.

Practical Implications

1. Prioritize Comparative Explanations in High-Risk Scenarios: Provide dual explanations by default—though it increases cognitive load, it is necessary.
2. Reassess Reasoning Traces: If displayed, add prompts (long reasoning ≠ correct reasoning).
3. Cultivate Users' Critical AI Literacy: Educate users to understand AI limitations and identify error patterns.
4. Establish Error Feedback Mechanisms: Such as confidence calibration and cross-validation with external knowledge.

Limitations and Future Directions

• Task Domain Limitation: Focused on logical verification tasks; subjective/complex domains (e.g., creative writing) need further validation.
• User Representativeness: Participants may not cover all groups; users from different backgrounds may react differently.
• Unknown Long-term Effects: The experiment measured immediate behavior; the adaptability of long-term use needs to be studied. Future research should explore the effects of explanations in more scenarios and users' long-term responses.