Some meaningful slice of the traffic hitting your server right now isn’t a person. It’s a training crawler, a search indexer, or an AI assistant fetching your page on a human’s behalf because someone asked it a question touching your content. Classic analytics wasn’t built to see any of this, so most site owners are blind on both the downside (unpaid scraping) and the upside (your page got cited and you’ll never know from a dashboard).
This post defines the field: what counts as AI traffic, why your analytics tool hides it, what’s actually known about how big it’s gotten, and the 8-class taxonomy we built to answer it properly.
What counts as AI traffic (crawlers vs indexers vs live agent fetches)
“AI traffic” gets used loosely. Worth splitting into the three things it actually means — they have almost nothing in common operationally:
- Training crawlers. Bots like GPTBot and ClaudeBot walk your site to build or refresh a model’s training corpus. No human waits on the other end — pure cost, bandwidth and server load with no attribution path back.
- Search/answer indexers. Bots like PerplexityBot and OAI-SearchBot crawl to build a live retrieval index that powers real-time answers, distinct from model training — the traffic most likely to translate into your content actually getting cited.
- Live agent fetches on behalf of a human. When someone asks ChatGPT or Claude a question and the assistant fetches your page right now to answer it, that request carries a user-agent like
ChatGPT-User— no browser, no referrer, but a real person waiting on the answer. This is the case that breaks the assumption baked into every referrer-based analytics tool: that a human visit always shows up as a browser navigation.
Vercel’s own breakdown also lands on three categories, cut slightly differently — training crawlers, live “grounding” fetches, and AI referrals (its third category is human click-throughs, not bot traffic) — and finds training crawlers the largest share by volume (Vercel blog). Each deserves different treatment: block one, welcome another, count the third — it looks like a bot but functions like a human visit.
Why GA4 can’t show you this
Google Analytics 4 automatically excludes “known bots and spiders” from every report, and this exclusion cannot be turned off by the property owner, per Google’s own support documentation (support.google.com/analytics/answer/9888366). Identification runs on “a combination of Google research and the International Spiders and Bots List,” maintained by the IAB — a generic list built to filter ad-fraud crawlers and legacy spiders, not to classify AI traffic. Google’s own page is explicit that you cannot see how much bot traffic was excluded or which bots triggered it. No toggle, no report, no volume number: GA4 quietly drops the rows.
Google did add something new: a native “AI Assistant” channel in GA4’s Default Channel Group, added May 13, 2026 and rolled out wider from June 7, 2026, per reporting on Google’s own changelog. When GA4 recognizes a referrer from a supported assistant, it tags the session medium = "ai-assistant". The covered list has changed since launch — and not by pure addition: Claude, named in the May launch list (alongside ChatGPT and Gemini), is absent from the current published definition, which names ChatGPT, Gemini, Copilot, Grok, and Deepseek. Perplexity traffic still lands in generic Referral unless you build a custom channel rule — and Claude referrals currently do too.
Even that new channel only catches referral clicks — a human clicking a link with the referrer intact. It cannot see crawlers (excluded by design) or live agent fetches, which carry no Referer header because the request originates server-side, independent of any browser navigation. A referrer-based tool can only count the subset of AI-driven traffic where a human clicked, the app didn’t strip the referrer, and the destination site correctly buckets that host. The crawl, the citation, the agent fetch that answered a question without anyone opening a browser tab — all invisible to it by construction. That’s the gap this category of tooling exists to close; we cover the ChatGPT-specific mechanics in how to track ChatGPT traffic.
The numbers: how big AI traffic has gotten
Treat all of the below as directional, not audited — vendor telemetry and single studies, and some figures conflict slightly by measurement window.
Cloudflare CEO Matthew Prince stated automated traffic reached roughly 57% of worldwide HTTP requests to HTML content, versus human traffic making up the rest, as of early June 2026 — reportedly the first time bots have outnumbered humans in Cloudflare’s data, per Search Engine Land’s reporting (two secondary write-ups differ on the exact decimal, 57.3% vs 57.5%, so “~57%” is the honest citation). That figure covers all automated traffic, not just AI — Cloudflare’s own numbers put AI crawlers specifically at around 20% of verified bot traffic as of May 2026, with AI-search bots adding a further few points, since most automated traffic is still non-AI (monitoring, scanners, legacy crawlers).
More useful for site owners: the crawl-to-referral ratio, how many pages an AI crawler fetches for every click-through it sends back. Cloudflare’s blog post breaking this down by purpose found, in an early-August 2025 snapshot: Anthropic’s crawlers at roughly 50,000 crawled pages per referral, OpenAI’s at 887:1, Perplexity’s at 118:1 (company-level ratios across each vendor’s crawler fleet) (blog.cloudflare.com) — training traffic near 80% of all AI crawler activity in that window. Independently, Fastly’s Q2 2025 Threat Insights Report (6.5 trillion monthly requests analyzed) found crawler-type bots (training and indexing crawls, in Fastly’s bucketing) at almost 80% of all AI-labeled bot traffic; the real-time fetcher bots (live ChatGPT/Perplexity fetches) were the smaller share by volume but the spikier one, hitting 39,000 requests per minute — enough, in Fastly’s words, to overwhelm unprotected origin infrastructure (fastly.com).
The consistent theme across Cloudflare, Fastly, and Vercel: training crawlers dwarf everything else by volume, and “AI reads your page” outpaces “AI sends you a human” by orders of magnitude — whether closer to 50,000:1 or 900:1 depends on crawler and window. A single aggregate number from GA4 (mostly zero, since bots are excluded) is worse than useless here; you need it broken down per bot, per class.
An 8-class taxonomy for real answers
We built our classifier around eight traffic classes instead of a binary human/bot flag, because “is this a bot” is the wrong question. The right one is “what kind of non-human, and does it matter to me”:
| Class | What it is | Example agents |
|---|---|---|
user |
A real human in their own browser | — |
ai_user_action |
Live AI browsing on a human’s behalf, in response to their question | ChatGPT-User, Claude-User, Perplexity-User |
ai_search |
AI search indexers building a live retrieval index | PerplexityBot, OAI-SearchBot |
ai_training |
AI training crawlers building or refreshing a model corpus | GPTBot, ClaudeBot, CCBot, Bytespider |
search_index |
Classic search engine crawlers | Googlebot, Bingbot |
social_unfurl |
Link-preview bots generating a chat/social card | Slackbot, LinkedInBot, Twitterbot |
scanner |
Security, uptime, and performance monitoring | — |
bot_other |
A recognized bot with no dedicated bucket, or a caught spoofer | UA claims GPTBot from a cloud provider’s or a non-AI vendor’s IP |
Six of those agents get their own deep-dive if you landed here after seeing one in your logs and googling the name: ClaudeBot, GPTBot, PerplexityBot, OAI-SearchBot, CCBot, and Bytespider.
The taxonomy is Cloudflare-compatible in structure — the same underlying split between training, search, and user-action bots — but it’s a first-class field in our own event schema, not an add-on report.
Why ai_user_action is the class to watch
Of the eight, ai_user_action most changes how you should think about “traffic.” Every other AI class is either pure cost (ai_training) or a proxy for future visibility (ai_search). This one is neither — it’s a human, right now, getting an answer grounded in your page, mediated through an assistant instead of a browser. Cloudflare’s August 2025 breakdown found ChatGPT-User responsible for “nearly three quarters” of request traffic within the user-action category, with Perplexity-User showing a similar pattern — thin relative to training-crawler volume, but concentrated in a couple of dominant agents.
It’s also the class GA4 structurally cannot separate out, even with its new AI Assistant channel, since there’s no browser navigation and no referrer to classify. If you only track referral clicks, this traffic either doesn’t exist in your reports or gets miscounted as direct — the difference between “AI ignores us” and “AI answers questions about us constantly, we just never saw it.”
If you want to see this breakdown on your own site, the traffic_class_breakdown tool returns exactly this table for any period you ask about — ask Claude “how much of my traffic last month was ai_user_action vs ai_training” and it calls the tool directly.
Detecting spoofers
None of this is worth anything if a scraper can just put GPTBot in its user-agent string and get bucketed as legitimate. Classification has to go past string-matching.
Our classifier starts from a curated list of 98 user-agent patterns, sourced from the community ai.robots.txt list, Anthropic’s and OpenAI’s own bot docs, and Cloudflare Radar — but a UA string alone is a claim. We verify it where the operator supports verification. For OpenAI’s bots that means checking source IPs against published IP ranges (OpenAI publishes gptbot.json with the blocks GPTBot crawls from). For Anthropic-style UAs that aren’t in the curated pattern list yet, a reverse-DNS fallback — the requesting IP’s reverse-DNS record has to resolve to an anthropic.com hostname, forward-confirmed — catches new crawler names before we’ve curated them; ingesting Anthropic’s newer published IP list is queued (details and the current gap disclosed in the ClaudeBot post).
A request claiming GPTBot from a cloud-provider IP, or from a range published by a non-AI vendor (Googlebot’s, say), gets demoted to bot_other — with one honestly-stated boundary: the check compares class families rather than companies, so a range belonging to a different AI vendor currently counts as agreement and isn’t demoted. That decision is why the taxonomy answers “should I block this,” not just “how much AI traffic do I get” — you can see spoofed-crawler volume separately from the real thing, and block genuine training crawlers via robots.txt without also blocking whatever’s impersonating them (a robots.txt block on GPTBot wouldn’t stop an impersonator anyway — it only binds bots that choose to obey it).
Monitoring AI traffic with an AI agent
The other reading of “an AI agent tool for website traffic monitoring” is literal: using an MCP-connected assistant to watch your own stats instead of opening a dashboard. It’s a genuinely new, still-small pattern — MCP servers for analytics show up on directories like MCP Market and Glama, and explainers from MotherDuck and Google Cloud describe the shape (pageviews, referrers, custom events as callable tools) — but too early a category for reliable adoption numbers, so we won’t invent one. What we can describe concretely is our own implementation, since it’s the whole product: mcp-analytics has no dashboard by design. You connect Claude, ChatGPT, or Cursor over MCP with OAuth 2.1 and PKCE, and ask questions directly.
Useful starting questions once connected: “break down last week’s traffic by class,” “which AI crawlers hit us most this month,” or “compare ai_user_action traffic this month to last.” Those map to traffic_class_breakdown, top_user_agents, and compare_periods — three of the 23 tools in the full catalog.
Where to start
If you’ve never looked at this split before, connect an MCP client and run traffic_class_breakdown for the last 30 days — that single call tells you whether AI traffic is a rounding error or a meaningful share of your logs. One setup note first, because it’s exactly the kind of blind spot this post is about: the default JS snippet only counts clients that execute JavaScript, and the crawler classes (ai_training, ai_search, most of ai_user_action) never do — capturing them takes the server-side ingest (a Rack middleware that isn’t on RubyGems yet as we publish this; the install steps already sit on your settings page, waiting on the gem’s public release) running alongside the snippet. Humans via snippet, bots via middleware, no double-counting. If a specific bot name showed up in your logs, the crawler posts cover individual agents, and /ai-crawler-index will aggregate classification data across contributing sites — its numbers are illustrative until 50+ sites contribute, as the page itself discloses.
Free tier: 100,000 hits/month, unlimited sites, no credit card — enough to run this analysis today: sign up.