Claude

Claude is Anthropic's model family, which added a web-grounded search mode that retrieves live pages and cites them alongside a generated answer, structurally similar to ChatGPT Search. Anthropic runs two separate crawlers for it: ClaudeBot collects training data, while Claude-SearchBot builds the separate index that Claude's search feature actually cites from.

Why it matters

Because ClaudeBot and Claude-SearchBot serve different purposes, a site can block one without affecting the other, so a robots.txt written only with model-training opt-outs in mind can unintentionally leave Claude's citation crawler blocked too, or vice versa. Tracking Claude as its own answer engine, rather than folding it into a generic LLM category, matters because its citation behavior and crawler rules don't move in lockstep with OpenAI's or Google's.

Related terms

• Large Language Model (LLM)

  A large language model (LLM) is an AI model trained on vast amounts of text to understand and generate human language. It's the underlying technology behind ChatGPT (OpenAI's GPT models), Claude (Anthropic), and Gemini (Google), and what every generative engine is built on top of.

• Retrieval-Augmented Generation (RAG)

  Retrieval-Augmented Generation (RAG) is the architecture behind most AI search: the system retrieves a set of relevant documents or passages at query time, then generates its answer grounded in that retrieved material, rather than purely from what it learned during training.

• Generative Engine

  A generative engine is an AI system, such as ChatGPT, Perplexity, or Google's AI Overviews, that answers a query by synthesizing one direct response from multiple retrieved sources, instead of returning a ranked list of links for the user to click through themselves.