What is GPTBot?
GPTBot is OpenAI’s web crawler. Per OpenAI’s own docs, it “is used to crawl content that may be used in training our generative AI foundation models” — a training-data crawler, full stop, distinct from the two other OpenAI bots covered below. In our own traffic taxonomy it lands in the ai_training bucket, the same class as ClaudeBot, CCBot, and Bytespider: crawlers that build a corpus, not ones acting live on behalf of a person.
If GPTBot showed up in your server logs, that’s the reason: something crawled you for a future model, not a live ChatGPT user reading your page right now.
The user agent string
Quoted verbatim from OpenAI’s docs, which explicitly flag that the version number can change without notice:
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot
Match on the GPTBot token, not the /1.4 version suffix — that number is not a stable identifier.
GPTBot vs ChatGPT-User vs OAI-SearchBot
This is the part most “what is GPTBot” posts skip, and it’s the part that actually matters if you’re deciding what to block. OpenAI ships three separate bots, and they cover three unrelated jobs:
| Bot | Job | Our traffic class |
|---|---|---|
| GPTBot | Crawls content for training future foundation models | ai_training |
| ChatGPT-User | Fetches a specific page live, when a user’s ChatGPT prompt references it | ai_user_action |
| OAI-SearchBot | Indexes the web to power ChatGPT Search citations, without collecting training data | ai_search |
OpenAI’s docs are explicit that “each setting is independent of the others” — you can allow one and disallow the other two in robots.txt, and OpenAI will honor each token separately. That independence is the whole point of a three-way split, and it’s also where most sites get confused: blocking GPTBot does nothing to your visibility in ChatGPT’s answers, because that’s OAI-SearchBot’s job, not GPTBot’s.
This is exactly the distinction GA4-style analytics tends to erase — it either lumps all three under “bot traffic” or drops them from the report entirely. We split them because the three have opposite implications for a site owner: one is a training opt-out, one is search visibility, one is direct answer-time fetches for a human who’s currently reading your content through ChatGPT.
GPTBot and robots.txt: the documentation
If you searched for the actual robots.txt syntax, here it is. The token and disallow line:
User-agent: GPTBot
Disallow: /
Per OpenAI: “Disallowing GPTBot indicates a site’s content should not be used in training generative AI foundation models.” That’s the entire scope of the directive — training-corpus opt-out, nothing else.
Official documentation: https://developers.openai.com/api/docs/bots. Note that this replaced the older platform.openai.com/docs/bots URL — if you have that link bookmarked, update it; developers.openai.com is the current path as of this writing.
Published IP ranges (JSON, machine-readable): https://openai.com/gptbot.json, live and returning HTTP 200 as of July 2026.
How to verify it’s really GPTBot
Anyone can set their User-Agent header to say “GPTBot.” A log line alone proves nothing. OpenAI’s documented verification mechanism is the IP range list above — cross-reference the source IP of the request against the CIDR ranges in gptbot.json. If the UA string says GPTBot but the request comes from an unrelated IP block (a random cloud provider, a residential ISP), it isn’t OpenAI.
Worth being precise about what OpenAI does not document: their bots page contains no mention of reverse-DNS or FCrDNS verification for GPTBot. Some third-party SEO blogs claim OpenAI crawlers also support PTR-based verification against an *.openai.com-style hostname, but none of those claims trace back to an OpenAI-authored source — treat that as unconfirmed, not as an official fallback if the IP-range check fails.
Our own classifier follows the IP-range approach rather than the DNS one for the OpenAI bots specifically: it fetches gptbot.json (plus searchbot.json and chatgpt-user.json) on a refresh cycle and checks incoming source IPs against those ranges. A UA that claims GPTBot but arrives from a cloud-provider IP — or from a range published by a non-AI vendor like Googlebot’s — gets demoted to bot_other rather than trusted at face value. One precision note on the limit of that check: it compares class families, not companies, so a range published by a different AI vendor currently counts as agreement and that particular spoof isn’t caught. It’s the same “don’t trust a single signal” logic we apply to any bot whose identity can be spoofed cheaply, with its current boundary stated honestly.
Should you block GPTBot?
Depends what you’re opting out of. Blocking GPTBot via robots.txt means: your content shouldn’t be used to train OpenAI’s future foundation models. It does not mean you disappear from ChatGPT Search citations (that’s OAI-SearchBot) and it does not stop ChatGPT from fetching your page live when a user pastes your URL into a chat (that’s ChatGPT-User). Those are independent switches — block all three, and OpenAI’s products stop touching your site in any capacity; block only GPTBot, and you keep your search-citation visibility while opting out of training.
On compliance: OpenAI’s own docs say GPTBot respects robots.txt. We didn’t find an independent third-party audit that specifically exercised GPTBot’s compliance the way Cloudflare’s August 2025 report did for Perplexity — that Cloudflare report did test OpenAI’s agent and, in its own words, found “ChatGPT-User fetched the robots file and stopped crawling when it was disallowed” — but it doesn’t appear to have separately verified GPTBot itself. So: OpenAI’s claim for GPTBot stands undisputed in what we could find, just less independently stress-tested than some of its peers.
On cadence: OpenAI doesn’t publish a stated crawl frequency for GPTBot, so we won’t invent one. If you want to know how often it actually hits your site, the honest answer is: check your own logs.
See GPTBot in your own traffic
Since mcp-analytics classifies GPTBot into ai_training automatically (cross-checked against gptbot.json whenever the source IP falls in any published range), you can ask directly in Claude rather than grep your access logs:
“Break down my traffic by class for the last 30 days — how much is ai_training vs ai_search vs regular users?”
That routes to our traffic_class_breakdown tool. For a raw list of which bots are hitting you and how often, top_user_agents gives you the ranked table, GPTBot included if it’s shown up.
One instrumentation caveat, in the spirit of this series: the default JS snippet only counts clients that execute JavaScript — GPTBot doesn’t. Capturing crawler hits takes the server-side ingest (a Rack middleware that isn’t on RubyGems yet as we publish this; the install steps already sit on your settings page, waiting on the gem’s public release) running alongside the snippet: humans via snippet, bots via middleware, no double-counting.
For the sibling bots in this series: what OAI-SearchBot is and how it differs, and what ClaudeBot is if you’re weighing a training opt-out across multiple vendors, not just OpenAI’s. The broader picture — how AI crawler traffic compares across all the classes we track — is in AI traffic analytics, and the aggregate cross-site view is planned at /ai-crawler-index — illustrative numbers today, real data once 50+ sites contribute.
If you’re currently doing this by hand in raw server logs: mcp-analytics is free up to 100,000 hits/month, no dashboard, just ask Claude.