Top AI crawlers, last 30 days

Hits = absolute count. Share = % of total crawler traffic (the 6.7% slice above), not % of all site traffic.

Bot Operator Hits Share of crawlers WoW robots.txt Note
GPTBot OpenAI 2,847 46.7% +18% GPTBot Training + ChatGPT browsing. Aggressive in May 2026.
ClaudeBot Anthropic 1,612 26.5% +22% ClaudeBot Training crawler. Polite, respects crawl-delay.
PerplexityBot Perplexity 802 13.2% +30% PerplexityBot Fastest-growing AI crawler in the current snapshot. Answer-grounding focus.
OAI-SearchBot OpenAI 412 6.8% +12% OAI-SearchBot ChatGPT Search indexing. Separate token from GPTBot.
CCBot Common Crawl 148 2.4% -5% CCBot Feeds many downstream LLMs. Monthly bulk crawl.
anthropic-ai Anthropic 92 1.5% +8% anthropic-ai Anthropic's older training-crawler UA. Distinct from ClaudeBot.
Bytespider Bytedance 71 1.2% -20% Bytespider Powers Doubao + TikTok search. Heavy traffic on some sites.
Applebot-Extended Apple 48 0.8% +15% Applebot-Extended Apple Intelligence training opt-out signal.
FacebookBot Meta 32 0.5% +3% FacebookBot Meta AI. Lower volume than GPT-class.
cohere-ai Cohere 19 0.3% ±0% cohere-ai Cohere's training fetch. Niche but present.
DuckAssistBot DuckDuckGo 11 0.2% +25% DuckAssistBot DuckDuckGo AI Assist. Just appeared in our logs this snapshot.

How to read a snapshot like this

OpenAI and Anthropic dominate. In a typical snapshot, GPTBot, ClaudeBot, OAI-SearchBot, and Claude-Web together would account for the large majority of AI crawler hits, with PerplexityBot among the fastest growers.

Bytespider cooled off. ByteDance's crawler was aggressive through 2024-2025; public reporting has it well below its peak since, so expect it mid-table rather than on top.

Newcomers show up as small, fast-rising rows. A new crawler like DuckAssistBot would first appear as a low-volume entry with a steep week-over-week climb. That pattern is the one to watch.

Common Crawl is lumpy. CCBot's volume is structurally bound to its monthly bulk-crawl cadence. Usually it's a single multi-day spike, not steady traffic.

Method

Hits classified via traffic_class_breakdown, our eight-class Cloudflare-compatible taxonomy. This index covers the ai_training bucket (GPTBot, ClaudeBot, CCBot, etc.) and ai_search bucket (PerplexityBot, OAI-SearchBot). Other classes that exist but aren't shown here: user (real humans), ai_user_action (Claude or ChatGPT fetching for a user), search_index (Googlebot, Bingbot), social_unfurl (Slackbot, LinkedInBot), scanner (uptime, security, headless monitoring), bot_other (recognised bot, no specific bucket).

User-agent matching uses a curated pattern list (~70 entries covering all major AI vendors and search crawlers) plus verification against the operators' published IP ranges (OpenAI, Google, Bing). Anthropic's crawlers are verified via FCrDNS instead: reverse-DNS plus forward-DNS round-tripping. Bots claiming to be GPTBot but routing from cloud IPs outside OpenAI's published ranges land in bot_other, not here.

What to do with this data

If you're a site operator:

  • Decide whether to Allow: or Disallow: each bot in your robots.txt. For an SEO/discoverability play, allow GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot. Those are the answer-grounding tokens. For an opt-out-of-training play, disallow them.
  • Notice if a crawler is over-aggressive (excessive hits, slow page-load impact). Use Cloudflare's bot management or your hosting platform's rate-limiting.
  • Cross-reference with our llms.txt explainer. Robots allow plus an llms.txt plus standard SEO is the complete AI-search stack.

Track your own crawler share

Sign up free at mcp-analytics.com, paste the tracking snippet on your site, and ask Claude: 

"How much of mysite.com's traffic is ai_training plus ai_search?"
"Show me top user agents in the ai_training class last 30 days."
"WoW change in GPTBot hits."

Free tier: 100,000 hits/month, unlimited sites, no card. Your contributions feed the aggregate index once we reach launch threshold. No identifiable per-site data is published.