From Eyeballs to Tokens: The Agentization Transformation of Web Architecture

This article discusses the paradigm shift in Web architecture—from the "eyeball economy" centered on human users' attention over the past three decades to the "token economy" adapted to the rise of AI agents. The core is that the Web needs to evolve from human-readable to machine-understandable, covering aspects such as architectural history, current agent-friendliness issues, solution principles, tool ecosystems, and future challenges.

1. Static HTML era (1990s-early 2000s): Document-based architecture with high readability but poor interactivity;
2. JS rise and DOM manipulation era (mid-2000s-early 2010s): AJAX and jQuery brought dynamic interactions, but code maintenance was difficult;
3. Component framework boom (2010s-early 2020s): React/Vue and other componentization + virtual DOM improved development efficiency, but JS volume was large, first-screen loading was slow, and it was unfriendly to crawlers/agents.

AI agents have become one of the main consumers of Web content, requiring structured and semantic data rather than visual presentation. Current Web problems:

1. Dynamic JS rendering leads to empty initial HTML, making it difficult for agents to extract content;
2. Pages are filled with noise such as ads/navigation, wasting AI token costs (70%-80% of costs are consumed by irrelevant content).

1. Revival of semantic HTML: Use tags like article/section + Schema.org JSON-LD markup;
2. Structured data and microformats: Embed metadata (e.g., product/article information) via JSON-LD/Microdata;
3. Lightweight rendering and edge computing: SSR/SSG/island architecture reduce client-side JS, and edge computing (Cloudflare Workers, etc.) enable low-latency dynamic content.

1. Content extraction tools: Firecrawl/Jina AI Reader and others extract clean content;
2. MCP protocol: A standardized interface proposed by Anthropic for agents to access resources uniformly;
3. Hybrid architecture: Progressive enhancement (core content in HTML, JS for enhancement), API-first, balancing human and machine needs.

1. Search ecosystem transformation: Generative Engine Optimization (GEO) focuses on content structure and understandability;
2. Privacy and security: Prevent malicious crawlers, sensitive information leakage, and implement machine verification mechanisms;
3. Technical debt migration: Adopt agent-friendly principles for new projects, and use tools to transition existing content. Conclusion: The Web will serve both humans and machines, building a more interconnected information ecosystem.