GPTBot

GPTBot is OpenAI's crawler for collecting web content used to train its models, operating separately from the crawlers OpenAI uses to power ChatGPT Search citations. A site can block GPTBot specifically to opt out of having its content used for model training, without affecting whether ChatGPT Search can still retrieve and cite that same content.

Why it matters

Because GPTBot and OAI-SearchBot serve entirely different purposes, teams often make an AI-crawler decision they didn't intend: blocking GPTBot to protect content from training use while assuming this also blocks ChatGPT Search citations, when the two are governed by separate robots.txt rules entirely.

Related terms

• OAI-SearchBot

  OAI-SearchBot is OpenAI's separate crawler that indexes web content specifically for ChatGPT Search, building the retrieval pool that feature cites from, independent of the GPTBot crawler used for model-training data collection. Blocking OAI-SearchBot removes a page from ChatGPT Search's citation pool without affecting training opt-outs.

• ChatGPT-User

  ChatGPT-User is the user-agent OpenAI applies when ChatGPT fetches a specific page live, in real time, at a user's explicit request within a conversation, distinct from the background indexing done by GPTBot or OAI-SearchBot. It represents an on-demand fetch rather than a systematic crawl.

• robots.txt (AI directives)

  robots.txt is the long-standing standard crawler-control file at a site's root, now extended with AI-specific user-agent rules that separately allow or disallow crawlers like GPTBot, ClaudeBot, and PerplexityBot. A single robots.txt file can permit standard search crawling while blocking one or more AI crawlers, or the reverse.

• ChatGPT Search

  ChatGPT Search is OpenAI's web-grounded answer mode inside ChatGPT, which retrieves current web pages before generating a response and displays a source panel alongside the answer. It launched as a standalone feature in late 2024 and uses a separate crawler, OAI-SearchBot, to build its citation index apart from GPTBot's model-training crawl.