ClaudeBot

ClaudeBot is Anthropic's crawler for collecting web content used in training its Claude models, operating separately from Claude-SearchBot, the crawler that builds the index Claude's web search and citation feature actually draws from. As with OpenAI's crawlers, blocking one does not automatically block the other.

Why it matters

A site owner who blocks ClaudeBot to opt out of AI training data collection needs to separately check Claude-SearchBot's access if the goal is also to remain visible in Claude's search feature, since the two crawlers are configured independently in robots.txt.

Anthropic's support page lists three agents: ClaudeBot collects content for model training, Claude-User fetches a page when someone asks Claude a question, and Claude-SearchBot analyzes content to improve search result quality. Anthropic says its bots honor robots.txt, support the Crawl-delay directive and do not try to bypass CAPTCHAs. It also warns that blocking Claude-User or Claude-SearchBot may reduce a site's visibility in Claude's answers to user searches.

A robots.txt rule that disallows ClaudeBot therefore opts a site out of training only. Log entries that claim to be ClaudeBot are not always genuine either: in late August 2026, GreyNoise found 824 IP addresses impersonating AI crawlers, ClaudeBot among them. Our guide to what each AI bot obeys in robots.txt covers the rules for every major crawler.

Related terms

• Claude-SearchBot

  Claude-SearchBot is Anthropic's crawler dedicated to indexing web content for Claude's search and citation feature, separate from ClaudeBot, which collects training data. It builds the pool of retrievable, citable pages Claude draws on when it grounds an answer in live web sources.

• Claude

  Claude is Anthropic's model family, which added a web-grounded search mode that retrieves live pages and cites them alongside a generated answer, structurally similar to ChatGPT Search. Anthropic runs two separate crawlers for it: ClaudeBot collects training data, while Claude-SearchBot builds the separate index that Claude's search feature actually cites from.

• robots.txt (AI directives)

  robots.txt is the long-standing standard crawler-control file at a site's root, now extended with AI-specific user-agent rules that separately allow or disallow crawlers like GPTBot, ClaudeBot, and PerplexityBot. A single robots.txt file can permit standard search crawling while blocking one or more AI crawlers, or the reverse.

• GPTBot

  GPTBot is OpenAI's crawler for collecting web content used to train its models, operating separately from the crawlers OpenAI uses to power ChatGPT Search citations. A site can block GPTBot specifically to opt out of having its content used for model training, without affecting whether ChatGPT Search can still retrieve and cite that same content.