llms.txt

llms.txt is a markdown file placed at a website's root that summarizes its content and structure specifically for AI systems and their crawlers, functioning as the closest thing this space has to an emerging technical standard. It gives an AI system a curated map of a site instead of forcing it to infer structure from a full crawl.

Why it matters

Adopting llms.txt early is a low-cost signal that a site is actively organizing itself for AI retrieval, even though adoption among AI crawlers themselves is still inconsistent and unconfirmed by any major platform as a ranking factor. It's worth implementing as good practice and a hedge, but it should never substitute for the structured data and content quality AI systems are confirmed to use.

Related terms

• llms-full.txt

  llms-full.txt is an expanded version of llms.txt that includes a site's full page content, rather than short summaries, in one consolidated markdown file intended for AI systems to ingest directly. It trades the compact, curated nature of llms.txt for completeness, at the cost of a much larger file.

• robots.txt (AI directives)

  robots.txt is the long-standing standard crawler-control file at a site's root, now extended with AI-specific user-agent rules that separately allow or disallow crawlers like GPTBot, ClaudeBot, and PerplexityBot. A single robots.txt file can permit standard search crawling while blocking one or more AI crawlers, or the reverse.

• Schema Markup / Structured Data

  Schema markup, or structured data, is machine-readable metadata embedded in a page, most commonly as JSON-LD, that explicitly tells crawlers what the content actually is: an article, an organization, a product, a set of questions and answers, rather than leaving that classification to inference.

• GPTBot

  GPTBot is OpenAI's crawler for collecting web content used to train its models, operating separately from the crawlers OpenAI uses to power ChatGPT Search citations. A site can block GPTBot specifically to opt out of having its content used for model training, without affecting whether ChatGPT Search can still retrieve and cite that same content.