Do Agents Need Semantic Metadata? A Comparative Study of Agentic Data Retrieval

This study focuses on a core question in the LLM era: Do agents still need semantic metadata like schema.org? Comparative experiments revealed:

• Baseline agents can answer more questions (40% higher coverage) but frequently encounter 'last mile' failures;
• Semantic agents are 65.7% more accurate in retrieving actionable data;
• Conclusion: The structured ecosystem remains the cornerstone of reliable autonomous workflows.

Research paper link: http://arxiv.org/abs/2605.28787v1, published on May 27, 2026.

A Decade of Semantic Metadata's Contributions

For over a decade, semantic metadata (e.g., schema.org) has supported the FAIR principles:

• Findable: Makes data easily discoverable by search engines;
• Accessible: Standardized descriptions help machines obtain data;
• Interoperable: Unified formats enable data exchange between systems;
• Reusable: Rich descriptions facilitate data understanding and reuse. Tools like Google Dataset Search are built based on these metadata.

New Possibilities Brought by LLMs

LLM capabilities have changed the game:

• Understand unstructured text;
• Navigate complex websites;
• Reason and judge relevance. This raises the question: If agents can directly read web pages, do they still need to rely on the semantic metadata middle layer?

Comparison of Two Agents

Feature	Baseline Agent	Semantic Agent
Data Source	Billions of open web documents	90 million datasets annotated with schema.org
Retrieval Method	General web search + LLM understanding	Structured metadata index
Advantage Hypothesis	Wide coverage, flexible	High precision, directly operable

Evaluation Framework

Adopting the 'LLM-as-a-judge' process, mapped to FAIR principles:

1. Semantic relevance: Does the result match the query intent?;
2. Data accessibility: Can the data be actually obtained?;
3. Computational utility: Can the data be directly used for analysis?

Test Scenarios

Covers real data retrieval tasks, simulating actual work needs of agents.

Divergence of Two Paths

• Baseline Agent: Breadth-first, can answer 40% more questions but frequently fails at the 'last mile';
• Semantic Agent: Precision-first, 65.7% more accurate in retrieving actionable data, and more reliably returns FAIR-compliant datasets.

Baseline Agent's 'Last Mile' Dilemma

Common failure modes:

Failure Type	Proportion	Description
Prose pages	20.1%	Returns text descriptions without actual data
Portal landing pages	8.5%	Points to data portal homepage instead of specific datasets
Unavailable downloads	-	Finds description but cannot get the file

Semantic Agent's Precision Advantages

Indicator	Semantic Agent Advantage
Metadata-rich registry accuracy	+44.9%
Machine-readable download page accuracy	+46.6%
Overall FAIR-compliant dataset retrieval accuracy	+65.7%

Limitations of Baseline Agents

1. Webpage noise: Too much irrelevant content, making it hard for LLMs to filter precisely;
2. Lack of structure: No standardized descriptions, making it hard to judge if it is data;
3. Link maze: Data is buried under multiple layers of pages, leading to navigation difficulties;
4. Diverse formats: Finds data but the format is not suitable for direct use.

Advantages of Semantic Agents

1. Structured indexing: schema.org provides machine-friendly descriptions;
2. Direct positioning: Metadata points to data files, avoiding last mile failures;
3. Standardized formats: FAIR principles ensure interoperable formats;
4. Quality filtering: Registries have basic quality requirements.

Analogical Understanding

• Baseline agent: Flipping through books in a library, may find unexpected content but with low efficiency;
• Semantic agent: Using a catalog index, quickly locates exact resources but depends on catalog completeness.

Recommendations for Agent Developers

1. Hybrid strategy: Baseline exploration + semantic precise acquisition;
2. Prioritize structured sources: Choose semantically annotated data sources when reliability is important;
3. Handle the last mile: Add data extraction modules for baseline agents.

Recommendations for Data Publishers

1. Continue investing in schema.org;
2. Ensure machine readability: Provide direct download links and standardized formats;
3. Maintain FAIR compliance.

Implications for Platform Designers

1. The structured ecosystem remains the cornerstone;
2. Agent-friendly design: Make it easy for agents to find data;
3. Invest in metadata quality.

Conclusion

Although unstructured retrieval supports exploratory tasks, the structured ecosystem is still an indispensable foundation for reliable autonomous workflows. Each method has its advantages:

• Exploration phase: The breadth of baseline agents is valuable;
• Execution phase: The precision of semantic agents is more reliable.

Limitations

1. Focuses on scientific datasets, other fields may differ;
2. Based on specific agent implementations, results may vary with different implementations;
3. Webpage structure and metadata quality change dynamically.

Future Research Directions

1. Optimal combination of hybrid architectures;
2. LLM automatically generating schema.org descriptions;
3. Agents adaptively choosing retrieval strategies.