What is Bytespider?

Bytespider is a web crawler operated by ByteDance, the company that also owns TikTok. It’s reported to collect page content in support of ByteDance’s AI systems — most commonly cited as feeding Doubao, ByteDance’s chat assistant, along with related search and recommendation products. In our traffic classification it sits in the ai_training class, the same bucket as GPTBot, ClaudeBot, and CCBot.

One thing to say plainly before anything else: ByteDance has not published an official Bytespider documentation page, at least not one we could locate. OpenAI, Anthropic, Perplexity, and Common Crawl all maintain a first-party page describing their crawler’s purpose and robots.txt token — in most cases with a machine-readable IP list, and in OpenAI’s and Common Crawl’s case with the full UA string printed too (Anthropic, as our ClaudeBot post details, lists only the embedded contact address). Bytespider has none of that. Every fact below — including the UA string itself — comes from third-party aggregators and news coverage, not a ByteDance-authored source. You’re being asked to trust an unverifiable bot on someone else’s word.

The user agent string

The commonly cited string, repeated across enough independent aggregators that it’s probably right, but with no ByteDance primary source to check it against:

Mozilla/5.0 (compatible; Bytespider; spider-feedback@bytedance.com)

Minor punctuation and contact-address variants show up across sources, and a longer browser-style form with a Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 … prefix circulates just as widely as the compact one above. Unlike OpenAI’s and Anthropic’s docs, which explicitly warn “the version number may change” on a page they control, there’s no spec page for Bytespider to check the current string against. Treat this as “what’s commonly observed,” not “what ByteDance guarantees.”

The compliance problem

This is the reason people search for Bytespider in the first place, and it’s the one part of this article we want to be precise about, because the claims vary a lot in how well-sourced they are.

Well-corroborated but not pinned to one report: since 2023, multiple site owners and security researchers have described Bytespider fetching URLs explicitly disallowed in their robots.txt. We couldn’t trace this to a single named researcher with a firm publication date the way we could for some of the other bots in this series — it’s a claim repeated across many sources rather than one detailed forensic writeup. Take it as “widely reported,” not “rigorously audited.”

The clearest data-backed report we found: HAProxy published a 2024 analysis of traffic to its own website, haproxy.com (measured via HAProxy Edge), stating that nearly 90% of the AI-crawler traffic it observed there came from Bytespider, and characterized a substantial share of it as ignoring Disallow rules. One site’s logs, not an industry aggregate — but still the strongest single data point behind Bytespider’s reputation: a specific vendor measuring specific traffic, not a general impression.

A more recent, less certain data point: a June 2026 third-party “monthly AI crawler report” claims Bytespider’s share of tracked AI-crawler traffic has since fallen — from roughly 10.1% to 7.3%, as ClaudeBot’s share grew. We’re citing this only as one analytics vendor’s estimate with unverified methodology, not a settled fact. It shouldn’t be read as ByteDance “getting compliant”; it’s a relative-share number from a source we can’t fully vouch for.

We found no public ByteDance response or rebuttal to any of these reports.

Why you can’t verify Bytespider

For GPTBot, ClaudeBot, OAI-SearchBot, PerplexityBot, and CCBot, there’s at least one authoritative way to double-check that a hit claiming to be that bot actually is: a published IP-range JSON file (OpenAI, Perplexity, Common Crawl, and — since 2026 — Anthropic all ship one), or documented reverse-DNS verification (Common Crawl).

Bytespider has neither. No published IP ranges, no documented reverse-DNS scheme, nothing we could find that lets you confirm a request’s origin against ByteDance’s own infrastructure. The UA string is the only signal available — and the easiest one to fake. Our own classifier (ingestion/internal/classify/source.go) has IP-based verification wired up for OpenAI, Google, and Bing; there’s no equivalent entry for ByteDance, because there’s nothing to verify against. A hit that says “Bytespider” is classified on that string alone — unless the source IP lands inside a cloud provider’s or a non-AI vendor’s published range, in which case the mismatch demotes it to bot_other (more on that below; a range belonging to a different AI vendor currently counts as agreement, since the check compares class families rather than companies). What’s impossible is the positive confirmation: there’s no ByteDance-published range to check a plausible-looking hit against.

Practically, anyone can send requests with the Bytespider UA and there’s no way — for us, for you, or apparently for ByteDance’s own reported crawler — to prove it’s genuine ByteDance infrastructure on the other end. Combine that with the compliance reports above and you get the honest summary: Bytespider is the bot in this series you have the least ability to verify or hold accountable.

Should you block it?

The documented mechanism is the same as any other crawler:

User-agent: Bytespider
Disallow: /

This is a community convention, not something ByteDance has confirmed it honors — there’s no official docs page to point to as the source of that token. Given the compliance reports above, the honest framing: robots.txt is the thing you’re supposed to do, but independent measurement suggests it isn’t reliably respected, and there’s no IP range or reverse-DNS check to confirm who’s actually knocking either way.

If robots.txt alone isn’t holding, the fallback is server- or edge-level UA blocking — reject or challenge requests whose UA matches Bytespider at your reverse proxy, CDN, or WAF, rather than relying on the crawler to self-police. That’s a blunter tool (it also catches anything spoofing the same string, which cuts both ways: you stop real Bytespider traffic that ignores robots.txt, but can’t distinguish it from a spoofer at that layer). It’s the closest thing to a hard block available, since there’s no cleaner mechanism to reach for.

What we see in our logs

Honest answer: not enough to say anything yet. Our own site has only been collecting cleanly classified data for a few weeks — far too thin a base to confirm or contradict the share estimates above, and we’d rather say that plainly than dress a small sample up as an observed trend. Ask us again in a year.

If you want to check your own numbers instead of taking anyone’s word for it (ours included), that’s exactly what our traffic_class_breakdown and top_user_agents MCP tools answer — ask Claude “how much ai_training traffic came from Bytespider in the last 30 days” and get a number from your own site’s classified events, not someone else’s aggregate report. Instrumentation caveat: the default JS snippet only counts clients that execute JavaScript — Bytespider doesn’t. Capturing crawler hits takes the server-side ingest (a Rack middleware that isn’t on RubyGems yet as we publish this; the install steps already sit on your settings page, waiting on the gem’s public release) running alongside the snippet.

See Bytespider in your own traffic

This is really the case for the whole taxonomy: an unverifiable bot with a documented compliance problem is exactly where “trust the UA string” isn’t good enough, and “just check bot_other” isn’t either. Our classifier gives Bytespider its own ai_training bucket when the UA matches, and demotes anything spoofing it against a mismatched IP to bot_other — you can see both, separately, in your own account.

top_user_agents shows the exact strings hitting your site, traffic_class_breakdown rolls them up by class over time, and compare_periods lets you check how Bytespider traffic on your domain is actually trending, whatever the aggregate reports claim. If you’re mapping the whole AI-crawler landscape rather than one bot at a time, our what-is-ccbot piece covers Common Crawl’s crawler (also ai_training, but far better documented), and ai-traffic-analytics is the broader piece on tracking this category. The aggregate, cross-site view is building at /ai-crawler-index — illustrative numbers today, real data once 50+ sites contribute.

mcp-analytics is free up to 100,000 hits/month, no credit card, and the whole point is that questions like this get answered inside Claude rather than a dashboard you have to remember to check.