Applebot-Extended

Applebot-Extended is Apple's opt-out token, used in robots.txt, that lets a site exclude its content from training Apple Intelligence and Apple's other generative AI features, separate from the standard Applebot rules governing Apple's search and Siri-related indexing across the web.

Why it matters

As Apple Intelligence expands its role in Siri and on-device search, Applebot-Extended is a rule worth deliberately setting rather than leaving unconfigured, since an unconfigured directive defaults to allowing the training use rather than opting a site out of it.

Related terms

• Google-Extended

  Google-Extended is not a crawler itself, but a robots.txt token site owners use to opt specific content out of use by Gemini and Google's other generative AI products, separate from the standard Googlebot rules governing traditional search indexing. A page can stay fully indexed in Google Search while excluded from Gemini's training data.

• robots.txt (AI directives)

  robots.txt is the long-standing standard crawler-control file at a site's root, now extended with AI-specific user-agent rules that separately allow or disallow crawlers like GPTBot, ClaudeBot, and PerplexityBot. A single robots.txt file can permit standard search crawling while blocking one or more AI crawlers, or the reverse.