What is ClaudeBot?

ClaudeBot is Anthropic’s web crawler for training data. Per Anthropic’s own documentation, it “helps enhance the utility and safety of our generative AI models by collecting web content that could potentially contribute to their training.” In our traffic classification, that makes it ai_training — the same bucket as OpenAI’s GPTBot and Common Crawl’s CCBot: crawlers that fetch pages to build a corpus, not to answer a specific user’s question in the moment.

If ClaudeBot is turning up in your server logs, it’s not a user visiting your site and it’s not Claude fetching a page because someone asked it a question right now (that’s a different bot — see below). It’s Anthropic building or refreshing training data.

The user agent string

Anthropic’s support documentation lists the contact address that appears in the string (obfuscated in the page markup, so a plain text search may miss it). The full user agent, corroborated by secondary UA-string aggregators, is:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)

Treat the /1.0 version token the way you’d treat any vendor’s version number in a crawler UA: current as of when this was last checked, not a permanent contract. Anthropic can bump it without notice.

ClaudeBot vs Claude-User (vs other Anthropic UAs)

Anthropic runs (at least) three named, separately-documented crawlers, and confusing them is the most common mistake in this space:

  • ClaudeBot — training-data collection, covered above. ai_training.
  • Claude-User — fetches a page live when a person asks Claude a question that requires reading a specific webpage. This is Claude acting on a human’s direct instruction in real time, not a background crawl. We bucket this as ai_user_action — it counts as human attention in your traffic, the same way ChatGPT-User or Perplexity-User does.
  • Claude-SearchBot — indexes pages to improve Claude’s own search/answer feature. Distinct from training. Our classifier doesn’t have a dedicated pattern for it yet; Anthropic-verified crawler traffic without a user marker currently lands in ai_training.

Anthropic’s docs name the Claude-User token but, as of this writing, don’t print a literal user-agent string for it at all (for ClaudeBot they at least list the embedded contact address) — so we won’t quote one here as if it were official. Secondary sources vary, and some reference an older Claude-Web/1.0 string Anthropic’s current page doesn’t mention. If you want to identify Claude-User hits with confidence, matching the token substring (Claude-User) is more reliable right now than trusting any specific full string you find on a third-party aggregator.

You may also see anthropic-ai in your logs — an older training-crawler token that shows up in community UA lists but isn’t mentioned on Anthropic’s current support page. Treat it as legacy; we can’t confirm from official sources whether it’s still active in production traffic or purely historical.

Each of these three (four, counting the legacy one) is a genuinely separate signal, and — see below — each is independently controllable in robots.txt.

How to verify it’s really ClaudeBot

Anthropic publishes a machine-readable IP list at claude.com/crawling/bots.json. Their own words: “if a crawler has a source IP address on [this list], it indicates that the crawler is coming from Anthropic.” As of our July 2026 check, that file lists roughly twenty prefixes — mostly single /32 addresses on Google Cloud ranges, plus one /22 block. This is the officially documented way to confirm a hit is genuinely Anthropic’s, the same pattern OpenAI (gptbot.json) and Google use.

One thing worth being precise about: Anthropic’s support documentation does not describe reverse-DNS (rDNS/FCrDNS) verification anywhere — the published method is the IP list.

And full disclosure on our own pipeline, in the spirit of this series. Our classifier recognizes ClaudeBot by its UA pattern first. If the source IP falls inside a range some vendor or cloud provider publishes, the two signals have to agree or the hit is demoted to bot_other; if the IP is unknown to us, the UA is trusted. We do run a reverse-DNS check (PTR must land on an *.anthropic.com hostname, forward-confirmed back to the same IP), but it’s a fallback that only fires for Anthropic-looking UAs we haven’t curated yet — not for ClaudeBot itself.

What we do not yet do is cross-check against the bots.json list above (we ingest OpenAI’s published ranges; Anthropic’s list is newer and queued). Until it lands, there’s an awkward side effect we’d rather disclose than hide: ClaudeBot crawls from Google Cloud addresses, which our classifier currently only knows as generic cloud infrastructure — so a genuine ClaudeBot hit can be demoted to bot_other by the very spoof-protection meant to guard its bucket. Ingesting Anthropic’s list fixes that.

Should you block ClaudeBot?

Anthropic’s own framing: blocking ClaudeBot signals that future site content should be excluded from training datasets. Anthropic documents each of the three crawlers and its robots.txt token separately, each with its own purpose — which reads as independent switches: block ClaudeBot, and Claude-User’s live fetches and Claude-SearchBot’s indexing keep working. Worth flagging the sourcing precisely, though: unlike OpenAI, whose docs literally say “each setting is independent of the others,” Anthropic never states that independence explicitly. We’re inferring it from the per-token documentation.

That’s the real tradeoff: block ClaudeBot and you likely stay out of future training runs, while remaining fully fetchable when a Claude user asks about your page directly, and fully indexable in Claude’s own search results. For most sites that want AI-search visibility but not training inclusion, disallowing ClaudeBot alone (leaving Claude-User and Claude-SearchBot alone) is the precise way to express that preference:

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

Anthropic’s documentation states all three crawlers honor robots.txt, including the non-standard Crawl-delay directive if you want to throttle rather than fully block. We haven’t found an independent, large-scale third-party audit of Anthropic’s compliance (the kind Cloudflare published for Perplexity, for instance) — its crawlers have a comparatively low-controversy reputation, but that reads to us as an absence of scrutiny rather than a confirmed clean record.

How much does ClaudeBot crawl?

Anthropic doesn’t publish a crawl cadence — no “we recrawl monthly” statement exists in their docs. The only volume signal we found is third-party: one crawler-traffic report estimated ClaudeBot’s share surged roughly 66% to become the #2 AI crawler by volume, at around 20% of tracked AI-crawler traffic, as of June 2026. Treat that as one analytics vendor’s estimate, not an Anthropic-confirmed number — we’re citing it because it’s the only figure available, not because we’ve verified its methodology.

See ClaudeBot in your own traffic

ClaudeBot is bucketed as ai_training (with the verification caveat disclosed above), so you don’t have to grep log files to answer “how much is Anthropic actually crawling me” — ask it in Claude directly. With mcp-analytics connected as an MCP server, traffic_class_breakdown gives you the ai_training vs. ai_search vs. ai_user_action split, and top_user_agents breaks out ClaudeBot from Claude-User and Claude-SearchBot specifically. A question like “How many ClaudeBot hits did I get in the last 30 days, and how does that compare to GPTBot?” is a normal thing to ask once it’s connected.

One instrumentation caveat, because this series exists to point out blind spots: the default JS snippet only counts clients that execute JavaScript — ClaudeBot doesn’t. To actually capture crawler hits you need the server-side ingest (a Rack middleware that isn’t on RubyGems yet as we publish this; the install steps already sit on your settings page, waiting on the gem’s public release) running alongside the snippet. Humans come in via the snippet, bots via the middleware, no double-counting.

For the aggregate picture across sites, /ai-crawler-index is the planned home — illustrative numbers today, real data once 50+ sites contribute. And if you’re trying to make sense of the wider AI-crawler landscape rather than just this one bot, our AI traffic analytics piece and the GPTBot writeup cover the same ground for OpenAI’s crawler.