OAI-SearchBot is a crawler run by OpenAI. According to OpenAI, it is used to surface websites in search results in ChatGPT’s search features. It fetches pages so ChatGPT can link to them when it answers with web search.
At a glance
- Operator
- OpenAI
- Purpose
- surface websites in ChatGPT's search features
- robots.txt token
OAI-SearchBot- Respects robots.txt
- yes, blocked sites are not shown in ChatGPT search answers (per OpenAI)
- IP list
openai.com/searchbot.json
Last checked:
Controlling OAI-SearchBot with robots.txt
OAI-SearchBot follows robots.txt and has its own user-agent token. OpenAI states that the settings for its bots are independent of each other. A site can allow OAI-SearchBot and at the same time block GPTBot, which collects content for model training:
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /
According to OpenAI, sites that opt out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links. OpenAI says robots.txt changes take about 24 hours to be reflected in its systems.
| Bot | Operator | Purpose | robots.txt token | robots.txt according to the operator | IP list published? |
|---|---|---|---|---|---|
| Google-Extended | Controls Gemini training and grounding (no crawler of its own) | Google-Extended | robots.txt token only; fetching is done by existing Google crawlers | n/a (no user agent of its own) | |
| GPTBot | OpenAI | Training | GPTBot | Disallowing it signals content should not be used to train foundation models | yes (openai.) |
| OAI-SearchBot (this term) | OpenAI | Search (ChatGPT) | OAI-SearchBot | opted-out sites are not shown in ChatGPT search answers | yes (openai.) |
| ChatGPT-User | OpenAI | User-triggered fetch | ChatGPT-User | robots.txt rules may not apply, says OpenAI (user-initiated) | yes (openai.) |
| ClaudeBot | Anthropic | Training | ClaudeBot | yes, Anthropic says its bots honor robots.txt | yes (claude.) |
| Claude-SearchBot | Anthropic | Search (Claude) | Claude-SearchBot | yes, Anthropic says its bots honor robots.txt | yes (claude.) |
| PerplexityBot | Perplexity | Search (Perplexity), not used for model training | PerplexityBot | controllable via robots.txt; the docs contain no explicit compliance statement | yes (perplexity.) |
| Applebot-Extended | Apple | Controls AI training (does not crawl, says Apple) | Applebot-Extended | robots.txt token only, applies to content Applebot already crawled | n/a (no crawler of its own) |
| CCBot | Common Crawl | Open web crawl archive | CCBot | can be blocked via robots.txt, says Common Crawl | yes (index.) |
| Meta-ExternalAgent | Meta | Training foundation AI models, improving products | meta-externalagent | can be blocked via robots.txt, says Meta | not stated by the operator |
A selection of ten AI bots from major operators, not a complete list. All values come from each operator's own documentation (as of 8 October 2026). The table does not show whether a bot actually complies with robots.txt; it only reflects what operators state. User agents can be spoofed; published IP lists or reverse DNS identify genuine requests. Sources: OpenAI: Overview of OpenAI Crawlers; Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler? (updated 2026-04-07); Perplexity: Perplexity Crawlers; Google: Google's common crawlers (last updated 2026-07-14); Google: Overview of Google crawlers and fetchers (last updated 2026-06-12); Google: Verifying Google crawlers and fetchers (IP range files); Apple: About Applebot (2026-09-04); Meta: Meta Web Crawlers; Common Crawl: CCBot. Last checked: .
How does it differ from other OpenAI bots?
OpenAI operates several bots with separate jobs:
- GPTBot crawls content that may be used to train OpenAI’s foundation models. Blocking it signals that content should not be used for training. It does not affect ChatGPT search.
- ChatGPT-User fetches pages when a person triggers an action in ChatGPT or a custom GPT. OpenAI notes that robots.txt rules may not apply to these requests.
- OAI-AdsBot visits only pages submitted as ChatGPT ads to check them against ad policies.
Identifying and verifying OAI-SearchBot
In the server logs, the bot appears with a user agent like this (as of October 2026):
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot
Because the string starts like a regular Chrome browser, log filters should match the token OAI-SearchBot. User agents can be faked, so OpenAI publishes the bot’s IP ranges in searchbot.json. Comparing a request’s IP address with that list confirms whether it is genuine.
- The IETF publishes the Robots Exclusion Protocol as RFC 9309 on the Standards Track. It defines how crawlers interpret the rules in a robots.txt file.Source: rfc-editor.org
- OpenAI introduces web crawler permissions for AI. Site owners can use them to state whether their content may be used for AI training.Source: openai.com
- Google introduces Google-Extended. This robots.txt token controls whether content helps improve Bard and the Vertex AI generative APIs.Source: blog.google
- After Bard is renamed, Google's documentation lists Gemini Apps as a product affected by Google-Extended.Source: developers.google.com
- Apple introduces the foundation models behind Apple Intelligence. Publishers can opt out of training use, according to Apple's documentation via Applebot-Extended.Source: machinelearning.apple.com
- Cloudflare lets customers block AI bots, scrapers and crawlers with a single click.Source: blog.cloudflare.com
- With the SearchGPT prototype, OpenAI separates search from training. Sites can appear in search results even if they opt out of AI training.Source: openai.com
- Google adds Google-CloudVertexBot to its list of crawlers. It crawls sites at the owner's request to build Vertex AI Agents.Source: developers.google.com
- Based on publisher feedback, Google clarifies the description of the Google-Extended token.Source: developers.google.com
- Cloudflare proposes that bots and agents cryptographically sign their requests, based on HTTP Message Signatures (RFC 9421).Source: blog.cloudflare.com
- Cloudflare changes its default to block AI crawlers unless site owners decide otherwise.Source: blog.cloudflare.com
- Cloudflare launches pay per crawl as a private beta. Crawlers either pay per request or receive a 402 status code with a price.Source: blog.cloudflare.com
OpenAI only says “last summer” for its crawler permissions (post of May 7, 2024); the often-cited August 2023 date for GPTBot comes from press reports. ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, OAI-SearchBot, ChatGPT-User, Meta-ExternalAgent and CCBot are documented by their operators, but without a verifiable launch date. Last checked: .
Common mistakes
According to reports, a blanket firewall or CDN rule against “AI bots” can block all OpenAI crawlers at once. The site then disappears from ChatGPT search even though robots.txt allows access. Server logs show this. If requests from OpenAI’s published IP ranges only receive status codes such as 403, a block is likely.
Frequently asked questions
What is OAI-SearchBot? OpenAI’s crawler for ChatGPT search. It finds and fetches pages that ChatGPT can cite and link to in search answers.
Is OAI-SearchBot used for training? OpenAI lists training as the job of GPTBot. OAI-SearchBot is described only in connection with search.
Should I allow OAI-SearchBot in robots.txt? OpenAI recommends allowing it, and requests from its published IP ranges, if a site wants to appear in ChatGPT search results. Blocking it keeps the site out of ChatGPT search answers.
How long does a robots.txt change take to apply? According to OpenAI, it can take about 24 hours for its systems to adjust after a robots.txt update.
Sources
- Overview of OpenAI crawlers – OpenAI (fetched 2026-10-06)developers.openai.com
- OAI-SearchBot IP ranges (JSON) – OpenAIopenai.com
Last updated: