In-depth Study on Citation Hallucinations in Research Agents: When AI Fabricates Non-Existent References

A large-scale study found that 3-13% of citation URLs generated by commercial LLMs and in-depth research agents are fake links caused by AI hallucinations. The more an agent claims to be capable of "in-depth research", the higher the proportion of fake citations. Researchers have open-sourced the urlhealth tool, which can detect and correct this issue. Through self-correction experiments, non-resolvable citations can be reduced by 6 to 79 times, with the final proportion controlled below 1%. This article will delve into the background, research methods, core findings, and solutions to this problem.

Dual Functions of Citations

In academic writing and professional research, citations serve key functions:

• Credibility Support: Provides external evidence for arguments, allowing readers to verify accuracy
• Knowledge Tracing: Establishes a chain of knowledge dissemination and traces the origin of ideas
• Academic Integrity: Reflects respect for knowledge sources and is a basic norm of the academic community

Specificity of AI-generated Citations

AI-generated citations face special challenges:

• Scale Effect: Generates dozens of citations in seconds, making manual verification almost impossible
• Authority Illusion: Standard format and rich details easily give people a trustworthy impression
• Ambiguous Responsibility: Unclear attribution of responsibility for fake citations (model/user/developer)

Dataset Construction

The research team built two large-scale datasets:

• DRBench: 53,090 citation URLs covering multi-domain research questions, evaluating 10 models/agents
• ExpertQA: 168,021 citation URLs covering 32 academic fields (from computer science to theology)

Evaluation Methods

• URL Survival Detection: Automated tools check if URLs are accessible
• Wayback Machine Verification: For inaccessible URLs, check historical archives via Internet Archive
• Failure Classification: Classify non-resolvable URLs into three categories: hallucinated URLs (never existed), link rot (existed but now invalid), temporary failures (temporarily inaccessible)

Overall Statistics

• Hallucinated URL Proportion: 3% to 13% of citation URLs are fabricated by AI and never actually existed
• Non-resolvable URL Proportion: 5% to 18% of URLs are inaccessible (including hallucinations, rot, temporary failures)

In-depth Research Agents vs. Ordinary Search-enhanced LLMs

• In-depth Research Agents: Generate more citations (15-25 on average) but have higher hallucination rates (up to 13%)
• Search-enhanced LLMs: Generate fewer citations (5-10) with lower hallucination rates (3-7%)

Domain Differences

• High-reliability Domains: Business (5.4% non-resolvable), computer science (6.2%), engineering technology (6.8%)
• Low-reliability Domains: Theology (11.4%), philosophy (10.8%), history (9.7%)

Model Differences

• Fabricator Models: Almost all non-resolvable URLs are hallucinations; overconfident fabrication
• Mistake-maker Models: Most non-resolvable URLs are link rot; attempt to retrieve real sources

Mechanism of Hallucinated URL Generation

• Overgeneralization of Pattern Matching: Synthesizes seemingly reasonable strings based on URL patterns, which are not real links
• Confusion Between Authority and Authenticity: Thinks ideas should have literature support, generating citations that "should exist" instead of verifying
• Blurred Line Between Retrieval and Generation: Fills gaps with generated content when retrieval fails, leading to hallucinations

Distinction Between Link Rot and Hallucinations

• Link Rot: Reasonable URL structure, has archives in Wayback Machine, page is offline or URL changed
• Hallucinations: Unreasonable URL combination, no records in Wayback Machine, page never existed

Open-source urlhealth Tool

The research team released urlhealth, a Python tool with the following features:

• Bulk URL Check: Efficiently detects the survival status of a large number of URLs
• Wayback Integration: Automatically queries Internet Archive to distinguish between rot and hallucinations
• Classification Report: Generates failure classification reports
• Easy Integration: Used as a library or command-line tool

Self-correction Experiments

• Experimental Setup: Agent generates citations → urlhealth checks → feedback → correction → repeat
• Effect: Non-resolvable citations reduced by 6-79 times, final proportion <1%
• Capability Differences: Depends on the model's tool usage ability; some models can correct effectively

For Researchers/Students

• Never fully trust AI citations; treat them as candidates that need manual verification
• Prioritize open-access literature to avoid paywall URLs
• Cross-verify important information; do not rely on a single AI citation

For AI Developers

• Integrate verification mechanisms; increase latency but improve reliability
• Transparently label verification status so users know the credibility
• Clearly express uncertainty instead of fabricating

For Publishing/Academic Institutions

• Update citation norms for AI-assisted research and disclose usage
• Develop fake citation detection tools
• Include AI citation limitations in methodology courses

Authority Crisis

Traditional authority markers (standard-format citations, professional expressions) may be unreliable; AI can mimic surface features but fabricate content. Need to develop new evaluation capabilities: check if it is verifiable, if citations are real, and if evidence is sufficient

New Paradigm of Human-Machine Collaboration

AI generation + human/tool verification; urlhealth is a prototype: AI generates and filters, tools verify and correct

Technical Limitations

• Can only detect URLs, not verify content
• Cannot detect fabricated metadata (authors, titles, etc.)
• May trigger a technical arms race (AI generates harder-to-detect hallucinations)