site-md: Redesigning Website Content Delivery for the AI Era

site-md is a lightweight middleware solution for Next.js applications, addressing the core conflict of traditional websites in the AI era: humans need aesthetic HTML interfaces, while AI agents excel at parsing structured Markdown. It intelligently detects the source of requests and automatically returns the corresponding format—HTML for humans, Markdown for AI—enabling dual presentation of the same content with no invasiveness (no need to refactor existing code), balancing user experience and machine readability.

With the rapid development of LLMs and AI search, AI agents like Perplexity and ChatGPT have become important content consumers. Traditional website architectures face a conflict: providing only HTML makes parsing complex for AI crawlers; forcing pure text delivery sacrifices human reading experience. Developers have to make a difficult trade-off between user experience and machine readability.

The core idea of site-md is one content, two presentation modes: intelligently detect the request source and automatically return HTML (for human browsers) or Markdown (for AI agents). Its uniqueness lies in non-invasiveness—developers only need to add two files to implement dual-mode delivery without refactoring existing code, respecting the existing workflow.

site-md is based on intelligent HTTP request identification: it checks the User-Agent and Accept headers to determine the requester; for AI crawler requests, it renders React components into Markdown (preserving semantic structures like title hierarchies, lists, code blocks, etc., and avoiding interference from ads and navigation bars). Integration with Next.js is lightweight—middleware handles format conversion, page components remain unchanged, and existing data fetching patterns are not affected.

site-md is suitable for three types of scenarios:

1. Content websites/blogs: Long-form content can be more accurately understood and cited by AI;
2. Documentation/technical blogs: AI agents can directly obtain clean Markdown, improving answer accuracy;
3. E-commerce/product pages: Structured Markdown makes it easier for AI to extract product features, enhancing visibility in AI searches.

site-md coincides with the paradigm shift in SEO: traditional SEO focuses on Google rankings, while emerging AEO (AI Optimization) focuses on citations and recommendations from AI Q&A systems. AI search engines emphasize semantics and context, so content parseability is crucial. Proactively providing Markdown is a forward-looking AEO strategy—no need to wait for AI crawlers to evolve; directly provide the format they excel at.

Currently, site-md only applies to the Next.js ecosystem and is not directly compatible with frameworks like Vue or Angular, but its core idea can be ported to other tech stacks. In the future, it may give rise to new HTTP standards or content negotiation protocols (e.g., Accept: text/markdown becoming a standard header), allowing AI agents to explicitly express their content preferences.