Reasoning Consistency Scanner: A Tool for Detecting 'Discrepancy Between Reasoning and Answer' in Large Language Models

This article introduces the open-source tool Reasoning Consistency Scanner, which aims to detect inconsistencies between the Chain of Thought (CoT) reasoning process and the final answer in Large Language Models (LLMs). This tool helps identify the phenomenon where models 'say one thing but do another', improving the reliability and interpretability of AI systems, and is applicable to scenarios such as model evaluation, data cleaning, and prompt optimization.

The Chain of Thought (CoT) technique allows LLMs to display their reasoning process, improving accuracy and interpretability for complex tasks, but there is a hidden problem: the reasoning process may be disconnected from the final answer. For example, in math problems, the reasoning is correct but the answer is wrong, or in logic problems, option A is refuted but selected. This inconsistency is harmful: it misleads user decisions, interferes with model evaluation, and exposes deep behavioral biases.

Reasoning Consistency Scanner is an open-source tool developed by SilviaSantano. Core idea: The thinking process of a reliable AI system must be logically consistent with its conclusions. The goal is to automatically identify cases of reasoning-answer inconsistency, helping to discover model weaknesses, improve training data, or adjust reasoning strategies—more efficiently than manual checks.

RCS uses a multi-dimensional approach to detect inconsistencies:

1. Logical Implication Analysis: Extract the implicit conclusion from the chain of thought and compare it with the answer;
2. Sentiment and Position Alignment: Analyze the sentiment polarity of the chain of thought towards options in classification tasks and check consistency with the answer;
3. Numerical Calculation Verification: Extract the calculation process and intermediate results from the chain of thought and verify their match with the answer;
4. Contradiction Detection: Identify direct contradictions within the chain of thought or with the answer (e.g., the chain of thought negates X but assumes X is true).

RCS is applicable to multiple scenarios:

1. Model Evaluation: Reveal reasoning quality issues hidden by traditional accuracy metrics (e.g., guessing the correct answer but with wrong reasoning);
2. Training Data Cleaning: Identify samples where the chain of thought does not match the answer (labeling errors or low-quality synthetic data);
3. Prompt Engineering Optimization: Test different prompt templates to find more consistent reasoning-answer relationships;
4. Production Monitoring: Detect inconsistent cases in real time and trigger alerts to respond to model anomalies.

RCS faces the following challenges:

1. The ambiguity of natural language leads to subjective consistency judgments (e.g., the chain of thought implies a tendency but has no clear conclusion);
2. It relies on NLP technology and may make mistakes, requiring manual review of key cases;
3. Some inconsistencies may be harmless (e.g., choosing a less mentioned option after exploring multiple possibilities).

RCS reflects the community's deep thinking on interpretability: It is not enough to just display the reasoning process; it must be consistent with behavior. It reminds us that the chain of thought of LLMs may be a post-hoc explanation rather than a prior reasoning process. Understanding this is crucial for correctly interpreting model outputs and is a step towards building trustworthy AI.

RCS provides a practical tool to address the reasoning-answer inconsistency in LLMs, helping developers improve models, researchers understand behaviors, and users build accurate perceptions. As AI applications expand in key areas, ensuring reasoning reliability becomes increasingly important. In the future, we look forward to more similar work to drive AI from 'seeming to think' to 'truly thinking'.