Google-Extended

Google-Extended is not a crawler itself, but a robots.txt token site owners use to opt specific content out of use by Gemini and Google's other generative AI products, separate from the standard Googlebot rules governing traditional search indexing. A page can stay fully indexed in Google Search while excluded from Gemini's training data.

Why it matters

Because Google-Extended and standard Googlebot rules are independent, a site can deliberately stay visible in traditional Google Search results while opting out of Gemini specifically, which matters for any brand weighing AI-training exposure against AI-search visibility as separate decisions.

Google's list of common crawlers defines Google-Extended as a standalone product token with no HTTP user agent of its own. It controls whether crawled content may be used to train future Gemini models and for grounding in Gemini Apps and in Grounding with Google Search on Vertex AI, and Google states that it does not affect inclusion or ranking in Google Search. The page does not list AI Overviews among the products it covers.

Because the token has no user agent, any request that announces itself as Google-Extended is fake by definition: GreyNoise logged 263,849 such forged sessions over three weeks in 2026. Our guide to verifying AI crawlers explains how to tell real bots from impersonators.

Related terms

• Gemini

  Gemini is Google's family of AI models, powering both Google AI Overviews and a standalone chat product with its own citation behavior, plus a separate Deep Research agent mode that compiles longer, multi-source reports. Its retrieval is search-grounded: it connects a generated answer back to Google's own search index before applying a further citation filter on top.

• Search-Grounded Retrieval

  Search-grounded retrieval is Gemini's mechanism of connecting a generated answer back to Google's own search index before applying a further citation filter on top. Rather than retrieving from an independent web crawl, Gemini effectively inherits whatever Google Search itself already considers relevant and trustworthy for a query, then layers its own criteria on that base.

• robots.txt (AI directives)

  robots.txt is the long-standing standard crawler-control file at a site's root, now extended with AI-specific user-agent rules that separately allow or disallow crawlers like GPTBot, ClaudeBot, and PerplexityBot. A single robots.txt file can permit standard search crawling while blocking one or more AI crawlers, or the reverse.