Claim-Level Evidence Admissibility: Enabling More Reliable Structured Outputs from Large Language Models

This study proposes a claim-level evidence admissibility framework. To address the hallucination problem in structured outputs of large language models (LLMs), it establishes a fine-grained evidence evaluation mechanism by drawing on the concept of evidence admissibility from the legal field, significantly reducing hallucination rates and improving factual accuracy. The framework requires traceable evidence support for each generated claim, with the core lying in the fine-grained integration of generation and verification.

LLMs tend to produce unsubstantiated hallucinations when generating structured outputs such as JSON and knowledge graphs. Traditional mitigation methods (prompt engineering, RAG, fine-tuning) have issues like separation of generation and verification, and lack of fine-grained evaluation. The claim-level evidence admissibility framework aims to address this pain point and achieve more reliable structured outputs.

Definition of Claim-Level Evidence Admissibility

Drawing on the concept of legal evidence admissibility, it establishes strict evidence admission standards for each atomic claim, requiring traceable evidence support.

Key Points of Technical Architecture

1. Claim Decomposition: Split structured outputs into atomic claim units
2. Evidence Retrieval: Independently retrieve relevant evidence for each claim
3. Admissibility Evaluation: Evaluate evidence from three dimensions: relevance, reliability, and sufficiency
4. Structured Reconstruction: Retain claims that pass the evaluation and reconstruct the output

Multi-Dimensional Evidence Scoring

• Relevance: Consistency with the claim's subject
• Timeliness: Freshness of information
• Source Credibility: Authority of the source and historical accuracy
• Logical Sufficiency: Whether the evidence supports the claim's conclusion

Dynamic Threshold Mechanism

Adjust admission standards according to claim type and domain (e.g., higher thresholds for medical claims).

Evidence Chain Traceability

Each accepted claim is accompanied by a complete evidence chain (source document, paragraph position, confidence level), suitable for high-risk scenarios.

Experimental Results

• Hallucination rate reduced by 40-60%
• F1 score for knowledge extraction increased by 15-25%
• JSON syntax correctness rate over 99%
• Over 85% of claims have evidence support

Application Scenarios

• Knowledge graph construction: Filter unsubstantiated entity relationships
• Database record generation: Prevent incorrect data from being stored
• Intelligent Q&A: Label evidence sources for answers
• Content moderation: Assist in identifying unsubstantiated claims

Current Limitations

• Computational overhead: Fine-grained retrieval increases inference costs
• Dependence on the quality of the underlying retrieval system
• Accuracy of evaluating complex composite claims needs improvement

Future Directions

1. Efficiency optimization: Approximate retrieval and caching strategies
2. Multimodal extension: Support for evidence from images, videos, etc.
3. Adversarial robustness: Identify maliciously constructed evidence
4. Domain adaptation: Refined domain-specific evidence standards

The claim-level evidence admissibility framework provides an important path for the reliable application of LLMs, emphasizing that generation capability and verification/traceability capability are equally important. For developers, integrating evidence awareness into system design is both a technical choice and a reflection of responsibility to users.

Project code and data can be found on GitHub: claim-level-evidence-admissibility