If OAI-SearchBot showed up in your logs and you’re about to add it to the same Disallow block as GPTBot: stop. It’s a different bot doing a different job, and blocking it has the opposite effect from what most people assume when they hear “OpenAI crawler.”
What is OAI-SearchBot?
OAI-SearchBot is OpenAI’s search-indexing crawler. In OpenAI’s own words, it “is used to surface websites in search results in ChatGPT’s search features” (developers.openai.com/api/docs/bots). The point of this article is what that means in contrast to GPTBot: OpenAI documents OAI-SearchBot separately from its training crawler, and its stated job is search visibility, not training-data collection. It builds the index ChatGPT’s search feature draws citations from.
In our own 8-class traffic taxonomy, OAI-SearchBot sits in ai_search — the same bucket as PerplexityBot: crawlers that index for AI answer engines rather than for classic search (search_index, Googlebot/Bingbot) or for model training (ai_training, GPTBot/ClaudeBot/CCBot).
The user agent string
Quoted verbatim from OpenAI’s docs:
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot
OpenAI’s own page notes the version number (1.4) may change without notice — match on the OAI-SearchBot token and the +https://openai.com/searchbot reference URL, not the trailing digits.
OAI-SearchBot vs GPTBot: not the same decision
This is the mistake we see most often: someone finds a “block AI crawlers” robots.txt snippet, applies it wholesale, and doesn’t realize they’ve just switched off three unrelated things at once.
OpenAI documents three separate crawlers, each independently switchable:
| Bot | Purpose (OpenAI’s wording) | Blocking it does |
|---|---|---|
| GPTBot | “used to crawl content that may be used in training our generative AI foundation models” | Opts your content out of future training corpora. Does not affect ChatGPT Search or live fetches. |
| OAI-SearchBot | “used to surface websites in search results in ChatGPT’s search features” — documented separately from GPTBot’s training crawl | Removes you from ChatGPT’s search-answer citations. Does not affect training-opt-out or live fetches. |
| ChatGPT-User | fetches a page live when a ChatGPT user’s query references it directly | Blocks that one specific live fetch. Does not affect training-opt-out or search indexing. |
OpenAI states this plainly: “Each setting is independent of the others” (developers.openai.com/api/docs/bots). Blocking GPTBot does not remove a site from ChatGPT Search citations, and blocking OAI-SearchBot doesn’t touch your training-data opt-out. If your goal is “keep my content out of training but still show up when someone asks ChatGPT a question,” the correct robots.txt has Disallow: / under User-agent: GPTBot and nothing under User-agent: OAI-SearchBot. Read our GPTBot piece if that’s the decision you’re actually trying to make.
How to verify it’s really OAI-SearchBot
Two signals, in order of trust:
- Published IP ranges. OpenAI publishes a machine-readable IP list at openai.com/searchbot.json (live and returning HTTP 200 as of July 2026). This is the mechanism OpenAI itself documents for confirming a hit is genuine — check the source IP against that list. Our own classifier does this: it’s one of the crawlers we pull IP ranges for on a 6-hour refresh (alongside GPTBot, ChatGPT-User, Googlebot, and Bingbot), and whenever a source IP falls inside any published range we track, the claimed UA has to agree with it.
- UA string match. Useful as a first filter, but on its own it’s weak — anyone can set a User-Agent header to anything.
What our classifier does with a mismatch, precisely (same boundary our sibling posts disclose): a UA claiming OAI-SearchBot from inside a cloud provider’s or a non-AI vendor’s published range gets demoted to bot_other; from an IP we know nothing about, the UA is trusted at face value — so a spoofer on a residential IP currently gets counted as the real thing, and a range published by a different AI vendor counts as agreement, since the check compares class families rather than companies. Note: OpenAI’s docs page doesn’t mention reverse-DNS verification anywhere — the IP-range JSON is the only officially documented method, so treat any “check the PTR record” advice you find elsewhere as unconfirmed.
Should you block it?
Short answer: probably not, unless you specifically want to be invisible in ChatGPT search results.
The tradeoff is narrow and one-sided compared to GPTBot. Blocking GPTBot is a legitimate, common choice — it’s a training-data opt-out, and reasonable people disagree about whether they want their content in a foundation model’s corpus. Blocking OAI-SearchBot is a visibility opt-out from a live product surface: if someone asks ChatGPT a question your page answers, you won’t be in the citation list. There’s no training-data angle here at all, by OpenAI’s own account.
If you do want to block it — maybe you don’t want ChatGPT driving any traffic, or you’re testing selective allowlisting — the robots.txt is the standard token, independent of the GPTBot line:
User-agent: OAI-SearchBot
Disallow: /
OpenAI states it respects robots.txt. We haven’t found an independent third-party audit that specifically tested OAI-SearchBot compliance the way Cloudflare’s 2025 report tested Perplexity’s crawlers, so treat that as OpenAI’s claim rather than an independently verified one — but we also haven’t seen anything contradicting it.
What OAI-SearchBot traffic predicts
Here’s the part worth paying attention to even if you never touch robots.txt: OAI-SearchBot hitting a page is a leading indicator, not just a crawl log entry. If your page is being indexed for ChatGPT search, it’s a candidate to actually get cited the next time a relevant query comes through. A page that never sees an OAI-SearchBot hit isn’t in the running.
We don’t have an official crawl-cadence figure from OpenAI to cite — no “crawls every N days” statement exists in their docs — so we can’t tell you how fresh the index stays. What we can say is that watching whether OAI-SearchBot visits a page at all, and which pages it prefers, tells you something about where you’re a candidate answer before you ever see a referral from chatgpt.com.
See OAI-SearchBot in your own traffic
Because mcp-analytics classifies OAI-SearchBot into ai_search automatically, you can ask about it directly instead of grepping server logs. In Claude (or any MCP client connected to your site), try: “Show me traffic_class_breakdown for the last 30 days — how much ai_search traffic did I get, and what does top_user_agents say about OAI-SearchBot specifically?” The traffic_class_breakdown and top_user_agents tools answer both halves of that in one pass.
Instrumentation caveat: the default JS snippet only counts clients that execute JavaScript — OAI-SearchBot, like every crawler in this series, doesn’t. Capturing its hits takes the server-side ingest (a Rack middleware that isn’t on RubyGems yet as we publish this; the install steps already sit on your settings page, waiting on the gem’s public release) running alongside the snippet: humans via snippet, bots via middleware, no double-counting.
If you’re trying to understand the bigger picture — which pages are showing up as AI-search candidates, and whether your content is even structured in a way that gets cited — our LLM SEO piece and how to track ChatGPT traffic cover the adjacent ground. The cross-site aggregate is planned at /ai-crawler-index — the numbers there are illustrative today and switch to real data once 50+ sites contribute, as the page itself discloses.
mcp-analytics runs entirely through MCP tools like these — no dashboard, ask Claude or ChatGPT directly. Free plan covers 100,000 hits/month with no credit card if you want to see your own OAI-SearchBot activity today.