A Framework for Verifying Reasoning Reliability of Large Language Models Without Retraining

Large language models (LLMs) face the 'hallucination' issue in reasoning tasks, generating seemingly reasonable but unsubstantiated reasoning processes. Traditional solutions like retraining are costly, prompt engineering has limited effects, and post-processing verification cannot fix structural errors in reasoning. A master's study from Stockholm University proposes a plug-in verification framework that improves reasoning reliability without modifying the model. Its core includes structured reasoning, task-adaptive verification, reasoning correction capabilities, and an explicit rejection mechanism. Experiments show this framework significantly boosts reasoning reliability, reduces erroneous outputs, and has low deployment costs and wide application scenarios.

The core problem of current LLMs in complex reasoning tasks is unreliable reasoning: answers may be correct but lack supporting reasoning, or fabricate explanations to back wrong conclusions—this is especially prominent in multi-step reasoning tasks.

Limitations of traditional solutions:

• Model retraining: High cost, requiring large volumes of labeled data
• Prompt engineering optimization: Limited effect and hard to generalize
• Post-processing verification: Cannot correct structural errors in reasoning

These methods either incur high costs or only address surface symptoms, failing to tackle the root cause.

The framework adopts a 'plug-in' architecture, adding an independent verification layer without modifying the underlying model. It undertakes four key responsibilities:

1. Structured reasoning: Convert input and reasoning into a graph structure where nodes represent reasoning steps/facts and edges represent logical dependencies, making reasoning relationships transparent.
2. Task-adaptive verification: Adjust verification strategies and strictness based on task types (logical reasoning, mathematical proof, etc.).
3. Reasoning correction capability: Targetedly correct reasoning using verification feedback, pointing out specific issues and guiding alternative paths.
4. Explicit rejection mechanism: When reasoning lacks sufficient support, the model explicitly refuses to answer to avoid unreliable conclusions.

The framework implementation includes four core components:

• Graph structuring module: Convert natural language reasoning into a graph structure, decomposing complex reasoning chains for easier analysis and verification.
• Verification rule engine: Built-in rules for logical consistency checks, fact support verification, reasoning chain integrity checks, cyclic dependency detection, etc.
• Revision strategy module: Analyze problem types (factual errors, logical loopholes, etc.) and propose targeted correction directions.
• Rejection decision maker: Combine verification results, confidence thresholds, task criticality, etc., to decide whether to refuse answering.

Researchers validated the framework's effectiveness on two datasets:

• NeuLR dataset: Focused on neuro-symbolic reasoning, significantly reducing unsupported reasoning while maintaining high task completion rates.
• CLUTRR dataset: Tested relational reasoning, effectively identifying weak links in reasoning.

Key findings:

• Significant improvement in reasoning reliability
• Rejection mechanism filters erroneous outputs, enhancing overall quality
• Deployment cost is far lower than model retraining
• Structured reasoning improves interpretability

The framework has application value across multiple fields:

• High-risk decision support: In scenarios like medical diagnosis and legal consultation, the rejection mechanism prevents misleading suggestions.
• Educational auxiliary tools: Ensure explanations are accurate and substantiated, avoiding the spread of incorrect knowledge.
• Research assistance: Help verify the reasoning chain of conclusions from AI-assisted analysis.
• Content review: Serve as an additional safeguard to ensure AI-generated content's reasoning withstands scrutiny.

Current limitations of the framework:

• Dependent on the completeness of built-in verification rules
• Additional verification steps increase reasoning latency
• New fields require adjustment of verification strategies

Future directions:

• Explore automated learning of verification rules
• Optimize verification efficiency
• Integrate the framework with more types of LLMs