Bytespider

Bytespider is ByteDance's web crawler, collecting training data across the web for its AI products, and it has a documented reputation for frequently ignoring robots.txt disallow rules that would normally block it, unlike most other major AI crawlers currently in operation.

Why it matters

Because Bytespider doesn't reliably respect robots.txt, blocking it there provides weaker protection than blocking most other AI crawlers, and a site with real concerns about ByteDance's use of its content may need server-level blocking rather than relying on the crawler directive alone.

Related terms

• robots.txt (AI directives)

  robots.txt is the long-standing standard crawler-control file at a site's root, now extended with AI-specific user-agent rules that separately allow or disallow crawlers like GPTBot, ClaudeBot, and PerplexityBot. A single robots.txt file can permit standard search crawling while blocking one or more AI crawlers, or the reverse.

• CCBot

  CCBot is the crawler behind Common Crawl, a large public dataset of web pages that isn't tied to any single AI lab but feeds model training for many different companies, making it one of the most consequential crawlers to account for even though it isn't operated by a household-name AI product.