robots.txt (AI directives)

robots.txt is the long-standing standard crawler-control file at a site's root, now extended with AI-specific user-agent rules that separately allow or disallow crawlers like GPTBot, ClaudeBot, and PerplexityBot. A single robots.txt file can permit standard search crawling while blocking one or more AI crawlers, or the reverse.

Why it matters

Because each AI lab runs multiple, purpose-specific crawlers, a robots.txt written with only one crawler in mind can unintentionally block a citation-relevant bot while leaving a training-only bot untouched, or vice versa. Auditing robots.txt against the full, current list of AI user-agents is a prerequisite for any GEO strategy.

Related terms

• GPTBot

  GPTBot is OpenAI's crawler for collecting web content used to train its models, operating separately from the crawlers OpenAI uses to power ChatGPT Search citations. A site can block GPTBot specifically to opt out of having its content used for model training, without affecting whether ChatGPT Search can still retrieve and cite that same content.

• OAI-SearchBot

  OAI-SearchBot is OpenAI's separate crawler that indexes web content specifically for ChatGPT Search, building the retrieval pool that feature cites from, independent of the GPTBot crawler used for model-training data collection. Blocking OAI-SearchBot removes a page from ChatGPT Search's citation pool without affecting training opt-outs.

• ClaudeBot

  ClaudeBot is Anthropic's crawler for collecting web content used in training its Claude models, operating separately from Claude-SearchBot, the crawler that builds the index Claude's web search and citation feature actually draws from. As with OpenAI's crawlers, blocking one does not automatically block the other.

• PerplexityBot

  PerplexityBot is Perplexity's crawler, responsible for indexing pages into the source pool its answer engine retrieves from and cites, functioning as the crawler equivalent of Perplexity's real-time, citation-first retrieval approach. Unlike some competitors, Perplexity runs one primary crawler rather than separating training and citation functions.

• llms.txt

  llms.txt is a markdown file placed at a website's root that summarizes its content and structure specifically for AI systems and their crawlers, functioning as the closest thing this space has to an emerging technical standard. It gives an AI system a curated map of a site instead of forcing it to infer structure from a full crawl.