ClaudeBot

ClaudeBot is Anthropic's crawler for collecting web content used in training its Claude models, operating separately from Claude-SearchBot, the crawler that builds the index Claude's web search and citation feature actually draws from. As with OpenAI's crawlers, blocking one does not automatically block the other.

Why it matters

A site owner who blocks ClaudeBot to opt out of AI training data collection needs to separately check Claude-SearchBot's access if the goal is also to remain visible in Claude's search feature, since the two crawlers are configured independently in robots.txt.

Related terms

• Claude-SearchBot

  Claude-SearchBot is Anthropic's crawler dedicated to indexing web content for Claude's search and citation feature, separate from ClaudeBot, which collects training data. It builds the pool of retrievable, citable pages Claude draws on when it grounds an answer in live web sources.

• Claude

  Claude is Anthropic's model family, which added a web-grounded search mode that retrieves live pages and cites them alongside a generated answer, structurally similar to ChatGPT Search. Anthropic runs two separate crawlers for it: ClaudeBot collects training data, while Claude-SearchBot builds the separate index that Claude's search feature actually cites from.

• robots.txt (AI directives)

  robots.txt is the long-standing standard crawler-control file at a site's root, now extended with AI-specific user-agent rules that separately allow or disallow crawlers like GPTBot, ClaudeBot, and PerplexityBot. A single robots.txt file can permit standard search crawling while blocking one or more AI crawlers, or the reverse.

• GPTBot

  GPTBot is OpenAI's crawler for collecting web content used to train its models, operating separately from the crawlers OpenAI uses to power ChatGPT Search citations. A site can block GPTBot specifically to opt out of having its content used for model training, without affecting whether ChatGPT Search can still retrieve and cite that same content.