What is CCBot?
CCBot is the crawler run by the Common Crawl Foundation, a nonprofit that has been building “an open repository of web crawl data that is universally accessible and analyzable by anyone” since long before “AI training data” was a phrase anyone searched for. Common Crawl itself doesn’t train a model — it publishes dated corpus snapshots that researchers and, notably, a long list of commercial LLM trainers reuse as a training-data ingredient. In our own classifier, CCBot is bucketed as ai_training alongside GPTBot and ClaudeBot (with a verification caveat we disclose below), but its shape is different from either: it’s not one company’s crawler, it’s the plumbing several companies’ crawlers draw from downstream.
That distinction matters for the rest of this article. Blocking GPTBot opts you out of OpenAI’s training pipeline specifically. Blocking CCBot opts you out of a shared public dataset that an unknown number of downstream consumers ingest wholesale — you can’t see the full list, and neither can Common Crawl.
The user agent string
Common Crawl documents the exact string on its own site:
CCBot/2.0 (https://commoncrawl.org/faq/)
Like every crawler UA covered in this series, treat the version number as a snapshot rather than a permanent identifier — vendors bump these without notice, and matching should key on the CCBot token, not the exact /2.0.
Who trains on Common Crawl data
Common Crawl’s own framing is that the corpus is “used for research” and reused as a training-data ingredient by other parties (commoncrawl.org/ccbot). It’s genuinely hard to name specific downstream models with primary-source confidence, because Common Crawl doesn’t publish a customer list and most LLM vendors don’t publish full training-data manifests. Our research turned up plenty of secondary commentary asserting that Common Crawl data has fed multiple well-known foundation models over the years, but nothing we’d trace back to a primary source we’d stand behind in print. So we’re deliberately not naming specific models here — a vendor’s own model card citing Common Crawl directly is the kind of source worth checking yourself, rather than trusting an aggregator’s paraphrase.
What we can say with more confidence: CCBot predates the current AI-training wave by years. The foundation was collecting web crawls long before “block AI crawlers” was a robots.txt conversation anyone was having. That history is part of why CCBot shows up in logs as a puzzle — site owners recognize GPTBot and ClaudeBot from recent coverage, but CCBot looks unfamiliar despite being older than both.
The crawl pattern: monthly bulk spikes
Unlike a training crawler that trickles requests continuously, or a live-fetch agent that only shows up when a user asks a question, CCBot’s traffic shape reflects Common Crawl’s operating model: periodic bulk crawls, historically run on a monthly cadence, each published as a dated snapshot. In practice that means CCBot traffic to a given site tends to look near-zero for weeks, then bursts of requests clustered in a short window, then quiet again until the next crawl. We’re hedging the exact current cadence deliberately — Common Crawl’s own page doesn’t state a specific interval as of this writing, and “monthly” is the crawl’s long-standing public reputation rather than a line we can quote verbatim from their site. Treat “roughly monthly, in bursts” as the reasonable working assumption, not a guaranteed schedule.
This bulk-spike pattern is the most useful diagnostic signal for telling CCBot apart from other AI training crawlers in your logs: flat-all-month-then-a-jump-for-a-few-days is a much better match for Common Crawl’s batch model than for a continuously-running commercial crawler.
How to verify it’s really CCBot
CCBot is one of the better-verifiable bots in this series, on two fronts. Common Crawl publishes a machine-readable IP list at https://index.commoncrawl.org/ccbot.json, and separately documents reverse-DNS verification directly on their own site: genuine CCBot requests resolve via PTR to a *.crawl.commoncrawl.org hostname (their published example: 18.97.14.84 resolves to 18-97-14-84.crawl.commoncrawl.org). Common Crawl explicitly notes this reverse-DNS check isn’t available over IPv6 (commoncrawl.org/ccbot).
Common Crawl is also unusually direct about the spoofing risk: their own docs warn that malicious crawlers falsely self-identify with the CCBot user agent, and tell webmasters to cross-check both the UA string and reverse DNS before trusting a hit as genuine. That’s exactly the failure mode our own classifier is built to catch generally — when a UA claims one crawler’s identity but the underlying signal doesn’t back it up, we demote the hit to a bot_other bucket rather than trusting the UA string alone. Honest limitation on our side — in both directions. This repo currently verifies CCBot by UA-pattern match only, with no automated IP-range or rDNS check wired in yet, so a well-crafted spoof from an unlisted IP could slip past our classifier the same way it could slip past a UA-only check on any other site. And the reverse failure exists too: Common Crawl crawls from AWS-hosted IP space, which our classifier only knows as generic cloud infrastructure — so until CCBot’s published ranges are wired in, a genuine CCBot hit can be demoted to bot_other by the very spoof-protection meant to guard its bucket. Our ClaudeBot and PerplexityBot posts disclose the same side effect for GCP- and AWS-hosted crawlers; ingesting the published ranges fixes it for all three.
Should you block CCBot?
This is the one where “just block it” is a weaker answer than it looks. Common Crawl’s own claim is that CCBot respects robots.txt, including the non-standard Crawl-delay directive, and we found no independent large-scale non-compliance report comparable to the Cloudflare-Perplexity or HAProxy-Bytespider findings covered elsewhere in this series — CCBot’s compliance reputation is comparatively uncontroversial.
The real tradeoff isn’t compliance, it’s what blocking removes. Standard opt-out:
User-agent: CCBot
Disallow: /
Because Common Crawl’s corpus is reused wholesale by a range of downstream consumers rather than feeding one company’s pipeline, blocking CCBot is a broader, blunter opt-out than blocking any single named crawler. You’re not just declining one vendor’s training run — you’re removing your site from a shared public dataset that some unknown number of research groups and commercial trainers draw from. If your actual goal is “keep my content out of commercial LLM training” specifically, CCBot is a defensible, high-leverage block. If your goal also includes “stay visible in AI search results” or “support open web research,” CCBot is a worse fit for a blanket block than a narrowly-scoped one, since you can’t selectively opt out of only the commercial reuse and keep the research reuse.
How much does CCBot crawl?
Beyond the monthly-batch shape described above, we don’t have a sourced figure for CCBot’s relative share of AI-crawler traffic the way some third-party vendor reports estimate for ClaudeBot or Bytespider. If you want a concrete number for your own site rather than an industry estimate, your own logs are the only trustworthy source — which is the point of the next section.
See CCBot in your own traffic
Since CCBot’s whole signature is the monthly-spike shape, the useful question isn’t “does CCBot visit me” but “when, and how much did that spike cost.” In Claude (or any MCP client connected to mcp-analytics), try: “Show me traffic_class_breakdown for ai_training this month, then compare_periods against last month — did CCBot’s hits spike?” The traffic_class_breakdown tool splits training, search, and live-fetch bots into separate classes (with the AWS-demotion caveat disclosed above — some genuine CCBot hits may currently sit in bot_other), and top_user_agents shows CCBot by exact UA string if you want to eyeball the pattern yourself.
Instrumentation caveat: the default JS snippet only counts clients that execute JavaScript — CCBot doesn’t. Capturing crawler hits takes the server-side ingest (a Rack middleware that isn’t on RubyGems yet as we publish this; the install steps already sit on your settings page, waiting on the gem’s public release) running alongside the snippet: humans via snippet, bots via middleware, no double-counting.
For the sibling posts in this series, see what GPTBot does and what Bytespider does — and if you’re weighing crawler traffic in general rather than one bot at a time, our AI traffic analytics piece and the aggregate AI crawler index are the wider view (the index shows illustrative numbers until 50+ sites contribute).