CCBot

CCBot is the crawler behind Common Crawl, a large public dataset of web pages that isn't tied to any single AI lab but feeds model training for many different companies, making it one of the most consequential crawlers to account for even though it isn't operated by a household-name AI product.

Why it matters

Because Common Crawl's dataset is reused across dozens of AI labs, a single CCBot block decision has outsized reach compared to blocking any one company's dedicated crawler, affecting a much wider and less visible set of downstream AI systems at once.

Related terms

• Bytespider

  Bytespider is ByteDance's web crawler, collecting training data across the web for its AI products, and it has a documented reputation for frequently ignoring robots.txt disallow rules that would normally block it, unlike most other major AI crawlers currently in operation.

• robots.txt (AI directives)

  robots.txt is the long-standing standard crawler-control file at a site's root, now extended with AI-specific user-agent rules that separately allow or disallow crawlers like GPTBot, ClaudeBot, and PerplexityBot. A single robots.txt file can permit standard search crawling while blocking one or more AI crawlers, or the reverse.

• Large Language Model (LLM)

  A large language model (LLM) is an AI model trained on vast amounts of text to understand and generate human language. It's the underlying technology behind ChatGPT (OpenAI's GPT models), Claude (Anthropic), and Gemini (Google), and what every generative engine is built on top of.