Reading
The first systematic evaluation framework for the citation quality of LLM deep research agents reveals: even the strongest models have a factual accuracy rate of only 39-77%, and more retrieval does not mean more accurate citations.
Section 01
This article focuses on the citation reliability issue of LLM-driven deep research agents. The first systematic evaluation framework reveals: even the strongest models have a factual accuracy rate of only 39-77%, and more retrieval does not mean more accurate citations. This article will discuss from the background, evaluation framework, findings, analysis, and future directions.
Section 02
With the rise of tools like OpenAI Deep Research, LLM research assistants have changed the way information is obtained, being able to automatically generate reports with citations. However, a key issue has been overlooked: are these citations reliable? Existing evaluations either verify claims in isolation or trust the model's self-assessment, lacking systematic inspection of the citations themselves, leading users to easily mistake citations as representing credible content.
Section 03
The research team launched the first source code attribution evaluation framework, which extracts and evaluates inline citations in LLM-generated reports through a reproducible AST parser. The framework closes the verification loop, allowing evaluators to judge citation quality based on source materials. The evaluation dimensions include: 1. Link validity (verifying URL accessibility); 2. Content relevance (degree of topic alignment); 3. Fact-checking (consistency between source content and claims, the most critical dimension).
Section 04
Benchmark test results for 14 closed-source and open-source LLMs: 1. The strongest cutting-edge models have a link validity rate of over 94% and relevance rate of over 80%; 2. Factual accuracy rate is only 39-77%; 3. Less than half of open-source models can generate reports with citations in one attempt. Ablation studies show: when the number of tool calls increases from 2 to 150, the fact-checking accuracy rate of two cutting-edge models decreases by an average of about 42%, proving that more retrieval does not mean more accurate citations.
Section 05
Reasons for the disconnect between surface citation quality and factual reliability: 1. Training bias: models learn citation formats but do not master strict fact-checking; 2. Retrieval noise: a large number of retrievals introduce more irrelevant information, reducing accuracy; 3. Generation pressure: models tend to generate fluent text rather than strictly accurate reports.
Section 06
This evaluation framework provides a tool to solve the citation reliability problem. For users: the number of citations ≠ credibility; for developers: more strict fact verification mechanisms and intelligent retrieval strategies are needed. With the popularization of AI content, establishing a reliable citation verification mechanism is key to the health of the information ecosystem, and this research lays the foundation for this.
Keep going with more reads from the same topic.
Nornir MCP Server is an enterprise-level server based on the Model Context Protocol (MCP). It seamlessly integrates large language models (such as Claude) with the Nornir network automation framework, supporting natural language orchestration for multi-vendor network devices (Cisco, Arista, Juniper, etc.), and providing production-grade features like a dual-engine architecture (NAPALM + Netmiko), intelligent filtering, and a secure sandbox.
Bibliothèque Française LLM is a structured indexing and annotation project for French public domain literature designed specifically for large language models (LLMs). It integrates multiple authoritative sources such as DraCor, Common Corpus, and Wikisource, providing metadata indexing categorized by genre, author, and era, as well as in-depth annotations for dramatic texts (including characters, lines, stage directions, etc.). Its aim is to enable LLMs to efficiently read and understand classic French literary works.
Splinter is a minimalist, high-performance key-value (KV) and vector storage system enabling zero-latency inter-process communication via shared memory and atomic operations. With only 766 lines of core code, it supports millions of operations per second and 768-dimensional vector storage, offering a new architectural approach for local LLM inference and data-intensive applications.
Folkering OS is the world's first AI-native bare-metal operating system, entirely written in Rust no_std without relying on Linux, POSIX, or libc. It can generate commands from scratch, compile them into WASM, and run them in 10 seconds, achieving true self-evolution.