Introduction: The SEO Revolution in the AI Search Era

Traditional Search Engine Optimization (SEO) is undergoing a profound transformation. With the popularity of AI conversational systems like ChatGPT, Claude, and Perplexity, users' way of accessing information has shifted from "search-click-read" to "ask-get answer". This means websites not only need to be indexed by traditional search engines but also understood and cited by Large Language Models (LLMs).

llm-seo is an open-source Agent skill project designed specifically for this new era. It provides a systematic methodology to help developers and website owners optimize their content, making it easier for AI crawlers to discover, for agents to understand, and to get cited in AI-generated answers.

What is LLM SEO?

LLM SEO (Large Language Model Search Engine Optimization) is a new optimization strategy targeting AI search and agent discovery. Unlike traditional SEO that focuses on keyword density and backlinks, LLM SEO emphasizes:

• AI Crawler Friendliness: Ensure AI crawlers like GPTBot, ClaudeBot, and PerplexityBot can correctly crawl and understand website content
• Semantic Clarity: Use structured data and clear definitional language to help LLMs accurately understand the services or products offered by the website
• Citation Value: Create content formats that are easy for AI systems to cite and recommend
• Agent Discovery: Enable AI agents to automatically integrate and use the website's APIs or services through standardized discovery files

The core goal of traditional SEO is to improve a website's ranking in traditional search results, relying on factors like keyword density and backlinks. However, the rise of AI conversational systems has changed how users access information—users no longer need to click multiple links to read content; instead, they get integrated answers directly through questions. This shift requires website content not only to be indexed by traditional search engines but also to be effectively understood, cited, and even called as tools by LLMs. LLM SEO is exactly the new optimization strategy born to adapt to this change.

Phase 1: Core SEO Infrastructure

The starting point of any LLM SEO optimization is to ensure a sound basic SEO architecture. This includes:

robots.txt Optimization: Fine-tune control specifically for AI crawlers—allow mainstream AI crawlers like GPTBot, ClaudeBot, Claude-SearchBot, PerplexityBot, and OAI-SearchBot to access public pages while blocking them from indexing internal management pages.

Sitemap (sitemap.xml): Provide a clear navigation map for AI crawlers. Set priority 1.0 for landing pages, 0.8 for document pages, and include the /llms.txt file with priority 0.6.

Metadata Optimization: The <title> tag is the only metadata reliably accessible by most AI systems. Use descriptive, definitional language (e.g., "X is...") instead of marketing language.

Phase 2: LLM Text Files

The project introduces two dedicated files: llms.txt and llms-full.txt:

/llms.txt: A concise Markdown file (1-2KB) containing core product overview (features, use cases, developer links, pricing). The key section is "Instructions for LLMs" (inspired by Stripe, guiding AI on best practices for usage).

/llms-full.txt: A complete reference document including all features, API endpoints, MCP tools, SDK examples, etc. It is recommended to generate it dynamically from OpenAPI specifications/MCP registries to keep it in sync.

Phase 3: Structured Data (JSON-LD)

Adopt the "Triple Schema Stacking" strategy—each page contains multiple JSON-LD code blocks:

• Organization Schema: Company information, logo, URL
• SoftwareApplication Schema: App metadata, pricing, category
• FAQPage Schema: FAQ section (highly valuable for AI citations)
• WebSite Schema: Website-level information
• Speakable Schema: Mark 2-3 most important content paragraphs as priority for AI retrieval
• HowTo Schema: Tutorial/guide pages
• TechArticle Schema: Document pages

In addition, it is recommended to place a security.txt file (RFC 9116 standard) in the /.well-known/ directory.

Phase 4: Agent & API Discovery (Conditional)

If providing APIs/SDKs/MCP servers, focus on:

OpenAPI Specification Endpoints: Unauthenticated endpoints (e.g., /api/openapi/public), with rich semantic descriptions for each operation.

Agent Discovery Files:

• /.well-known/agent-card.json: A2A protocol metadata file (promoted by Google and Linux Foundation)
• /.well-known/ai-plugin.json: OpenAI plugin manifest (legacy format)

Registration & Indexing:

• MCP Registry: Register at registry.modelcontextprotocol.io
• PulseMCP/Smithery: List to expand discovery
• Context7: Submit to context7.com/add-library or add a context7.json file.

The final step of optimization is to establish a monitoring system. It is recommended to use Google Analytics 4 (GA4) to set up custom channel groups and track AI traffic sources, including platforms like chat.openai.com, chatgpt.com, perplexity.ai, claude.ai, and copilot.microsoft.com.

Mistake	Solution
Missing "Instructions for LLMs" section in llms.txt	Add a Stripe-style section to guide AI on best practices for usage
Static llms.txt out of sync with APIs	Generate dynamically from OpenAPI specifications/MCP registries
Blocking all AI crawlers in robots.txt	Allow access to public pages, block only private routes
Duplicate FAQ data in components and JSON-LD	Extract to a shared module and import in both places
Not setting metadataBase	Must set—required for OG/Twitter absolute URL combination
Missing Speakable Schema	Mark key content paragraphs as priority for AI retrieval
Only one JSON-LD block per page	Use triple schema stacking—multiple schemas per page
Not registered in MCP Registry/Context7	Register to maximize AI agent discoverability

The llm-seo project focuses on the following emerging standards:

• WebMCP: A W3C initiative (Google + Microsoft) that exposes structured tools to browser AI agents via navigator.modelContext. Chrome Canary already provides a preview, with native support expected in H2 2026.
• /.well-known/mcp.json: Automatic discovery of MCP server cards (SEP-1649, SEP-1960), to be implemented once the specification stabilizes.
• Arazzo Specification: Multi-step API workflow orchestration for complex agent integration.

As AI systems become the primary entry point for users to access information, LLM SEO is no longer an option but a necessity. The llm-seo project provides a comprehensive, actionable framework to help websites and developer tools maintain visibility and relevance in the new era.

By implementing the five-phase workflow, developers can ensure their products are not only discovered by traditional search engines but also understood, cited, and recommended by AI systems—this is the key to digital visibility in the future.