Cloudflare added an AI opt-out flag to robots.txt. Google says no crawler obeys it.

On September 24, 2025, Cloudflare added three new preferences to robots.txt, each one a plain yes-or-no flag called search, ai-input, and ai-train. More than 3.8 million domains turned the feature on through Cloudflare's own managed robots.txt tool within its first year. Ten months after launch, Google gave its verdict on what the flags actually do. In a Reddit reply reported in July 2026, Google's John Mueller said the directive has no effect on any crawler or large language model he's aware of, and that it mostly adds bloat to a robots.txt file. For a company that just told millions of site owners they finally had a say over AI access, that's an awkward review.

What the Content Signals Policy actually controls

Content Signals Policy is a preference, not a technical block. It lets a site declare three settings on a Content-Signal line in robots.txt: search, meaning a crawler may index the page and show a snippet; ai-input, meaning an AI model may pull from the page when answering a live question; and ai-train, meaning the content may be used to train or fine-tune a model.

Cloudflare releases the policy text itself under a CC0 license, and any site on its managed robots.txt now gets search=yes, ai-train=no by default, with no stance set on ai-input because Cloudflare says it doesn't know a given customer's preference for that one. The company built the feature after what its own announcement describes as hundreds of conversations with publishers who wanted something between letting a crawler take everything and blocking it outright. A site can already write a rule naming one crawler directly, something like User-agent: GPTBot or User-agent: PerplexityBot, but Content Signals was designed to work across every engine at once without naming a single company. That's also exactly why nothing forces any of them to read it. Our guide to what robots.txt actually blocks when AI crawlers visit a site covers how those older, bot-specific rules behave once a crawler shows up.

Why Google says none of it matters

Google hasn't built anything that reads the Content-Signal line, and according to one of its own engineers, neither has anyone else worth worrying about. Google's John Mueller told a Reddit thread that the directive, which he described only as something a CDN had made up, has no effect on any crawler or LLM he knows of.

It has no effects whatsoever for any crawler or LLM. It just adds bloat and future maintenance to your robots.txt file.

That's Google's John Mueller, answering on Reddit in July 2026 and hedging the claim with “as far as I know,” which leaves room for some crawler somewhere to be quietly honoring it anyway. But the practical message to site owners was blunt: setting ai-train=no does not stop Googlebot, and it almost certainly doesn't stop anyone else either, unless that company has separately and publicly chosen to respect it. It's a familiar shape of disappointment. Our look at why noindex doesn't block AI crawlers found the same gap with an older, more established tag: noindex reliably keeps a page out of Google's search index because Google built its indexing pipeline to honor it, but no AI crawler is contractually required to treat an AI-specific signal the same way, and plenty don't.

One company split its crawler instead of ignoring the question

Not every AI company treats granular permissions as pointless. Anthropic separated its web crawler into three distinct bots in February 2026, each with its own user-agent string and its own documented behavior, which is closer to what Content Signals was reaching for than Google's dismissal suggests the industry wants.

ClaudeBot still collects pages for model training. Claude-User fetches a page only at the moment a live Claude user asks about it. Claude-SearchBot indexes content for Claude's own search results, and Anthropic's support documentation warns that blocking Claude-SearchBot specifically can reduce a site's visibility and accuracy in those results. Blocking the training crawler doesn't touch the other two. That's a real version of the choice Content Signals tried to offer, just built with three named, independently verifiable user agents instead of one preference line nothing is obligated to check. It's also the kind of distinction that actually matters for a brand trying to show up in Claude's answers without having every page scraped for training, a trade-off we cover in blocking AI training can quietly block your AI citations too.

Cloudflare's next move: blocking crawlers that mix uses

Cloudflare didn't wait for AI companies to adopt Content Signals voluntarily. Starting September 15, 2026, its default settings began blocking what it calls mixed-use crawlers, bots that combine search indexing, live AI-agent access, and model training inside one user agent, from any page that carries ads.

The change applies by default to new Cloudflare customers, to new sites added by existing customers, and to every site still on Cloudflare's free tier, though any site owner can turn it back off. Cloudflare's named example is Googlebot, which crawls once for Search, AI Overviews, and AI Mode together, and the company claims Google ends up with roughly twice the access of any single AI competitor as a result, a comparison Google disputes while pointing to Google-Extended as the real opt-out for training and some AI products without losing Search inclusion. Cloudflare also says more than half of AI crawler traffic refetches pages that haven't changed since the last visit, which it frames as wasted publisher bandwidth rather than genuine demand for fresh content, according to TechCrunch's reporting on the change. Cloudflare's older Pay Per Crawl tool, which uses an HTTP 402 status code to charge crawlers per request, is evolving into a model it calls Pay Per Use, paying publishers only when their content shows up inside an AI product's actual output; Ceramic.ai and You.com are the first two partners. We've covered Cloudflare's AI crawler settings before, including how to check whether Cloudflare is already blocking your AI crawlers without you realizing it.

What this means if you're tracking AI visibility

None of this changes what actually gets a brand cited. It changes who can reach a site at all, and that's the layer underneath every AI Share of Voice number a company tracks. A flag in robots.txt that nothing reads is not a visibility strategy; it's a note nobody opens.

The more reliable signal is still direct observation. Checking server logs for which user agents actually show up, and how often they refetch pages that haven't changed, tells a brand far more than any preference line can, a method we walk through in our guide to the crawl-to-refer ratio. The same logic applies to llms.txt, a separate file some sites publish listing which pages an AI model should prioritize: like Content Signals, it only works for the engines that have chosen to read it, which is still a short list, as we cover in our complete guide to llms.txt. Anthropic's three-crawler split is still the only public example of a major AI company building that kind of granular, verifiable control into its own infrastructure rather than asking site owners to trust a preference file.