# OAI-SearchBot

Category: [AI Bots & Crawlers](https://seo-geo-glossary.com/en/glossary/category/ai-bots-crawlers/) › [OpenAI, Anthropic & Perplexity bots](https://seo-geo-glossary.com/en/glossary/category/ai-bots-crawlers/openai-anthropic-perplexity/) · Also known as: OAI SearchBot

**OAI-SearchBot** is a crawler run by OpenAI. According to OpenAI, it is used to surface websites in search results in ChatGPT's search features. It fetches pages so ChatGPT can link to them when it answers with web search.

## Controlling OAI-SearchBot with robots.txt

OAI-SearchBot follows robots.txt and has its own user-agent token. OpenAI states that the settings for its bots are independent of each other. A site can allow OAI-SearchBot and at the same time block GPTBot, which collects content for model training:

```robots
User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /
```

According to OpenAI, sites that opt out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links. OpenAI says robots.txt changes take about 24 hours to be reflected in its systems.

## How does it differ from other OpenAI bots?

OpenAI operates several bots with separate jobs:

- **GPTBot** crawls content that may be used to train OpenAI's foundation models. Blocking it signals that content should not be used for training. It does not affect ChatGPT search.
- **ChatGPT-User** fetches pages when a person triggers an action in ChatGPT or a custom GPT. OpenAI notes that robots.txt rules may not apply to these requests.
- **OAI-AdsBot** visits only pages submitted as ChatGPT ads to check them against ad policies.

## Identifying and verifying OAI-SearchBot

In the server logs, the bot appears with a user agent like this (as of October 2026):

```txt
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot
```

Because the string starts like a regular Chrome browser, log filters should match the token `OAI-SearchBot`. User agents can be faked, so OpenAI publishes the bot's IP ranges in `searchbot.json`. Comparing a request's IP address with that list confirms whether it is genuine.

## Common mistakes

According to reports, a blanket firewall or CDN rule against "AI bots" can block all OpenAI crawlers at once. The site then disappears from ChatGPT search even though robots.txt allows access. Server logs show this. If requests from OpenAI's published IP ranges only receive status codes such as 403, a block is likely.

## Frequently asked questions

**What is OAI-SearchBot?**
OpenAI's crawler for ChatGPT search. It finds and fetches pages that ChatGPT can cite and link to in search answers.

**Is OAI-SearchBot used for training?**
OpenAI lists training as the job of GPTBot. OAI-SearchBot is described only in connection with search.

**Should I allow OAI-SearchBot in robots.txt?**
OpenAI recommends allowing it, and requests from its published IP ranges, if a site wants to appear in ChatGPT search results. Blocking it keeps the site out of ChatGPT search answers.

**How long does a robots.txt change take to apply?**
According to OpenAI, it can take about 24 hours for its systems to adjust after a robots.txt update.

## At a glance

- **Operator:** OpenAI
- **Purpose:** surface websites in ChatGPT's search features
- **robots.txt token:** OAI-SearchBot
- **Respects robots.txt:** yes, blocked sites are not shown in ChatGPT search answers (per OpenAI)
- **IP list:** [openai.com/searchbot.json](https://openai.com/searchbot.json)
- **Docs:** [OpenAI: Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots)

Last checked: October 8, 2026

## AI bots compared

| Bot | Operator | Purpose | robots.txt token | robots.txt according to the operator | IP list published? |
|---|---|---|---|---|---|
| Google-Extended | Google | Controls Gemini training and grounding (no crawler of its own) | Google-Extended | robots.txt token only; fetching is done by existing Google crawlers | n/a (no user agent of its own) |
| GPTBot | OpenAI | Training | GPTBot | Disallowing it signals content should not be used to train foundation models | yes (openai.com/gptbot.json) |
| OAI-SearchBot | OpenAI | Search (ChatGPT) | OAI-SearchBot | opted-out sites are not shown in ChatGPT search answers | yes (openai.com/searchbot.json) |
| ChatGPT-User | OpenAI | User-triggered fetch | ChatGPT-User | robots.txt rules may not apply, says OpenAI (user-initiated) | yes (openai.com/chatgpt-user.json) |
| ClaudeBot | Anthropic | Training | ClaudeBot | yes, Anthropic says its bots honor robots.txt | yes (claude.com/crawling/bots.json) |
| Claude-SearchBot | Anthropic | Search (Claude) | Claude-SearchBot | yes, Anthropic says its bots honor robots.txt | yes (claude.com/crawling/bots.json) |
| PerplexityBot | Perplexity | Search (Perplexity), not used for model training | PerplexityBot | controllable via robots.txt; the docs contain no explicit compliance statement | yes (perplexity.com/perplexitybot.json) |
| Applebot-Extended | Apple | Controls AI training (does not crawl, says Apple) | Applebot-Extended | robots.txt token only, applies to content Applebot already crawled | n/a (no crawler of its own) |
| CCBot | Common Crawl | Open web crawl archive | CCBot | can be blocked via robots.txt, says Common Crawl | yes (index.commoncrawl.org/ccbot.json) |
| Meta-ExternalAgent | Meta | Training foundation AI models, improving products | meta-externalagent | can be blocked via robots.txt, says Meta | not stated by the operator |

Sources: [OpenAI: Overview of OpenAI Crawlers](https://developers.openai.com/api/docs/bots); [Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler? (updated 2026-04-07)](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler); [Perplexity: Perplexity Crawlers](https://docs.perplexity.ai/guides/bots); [Google: Google's common crawlers (last updated 2026-07-14)](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers); [Google: Overview of Google crawlers and fetchers (last updated 2026-06-12)](https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers); [Google: Verifying Google crawlers and fetchers (IP range files)](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests); [Apple: About Applebot (2026-09-04)](https://support.apple.com/en-us/119829); [Meta: Meta Web Crawlers](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/); [Common Crawl: CCBot](https://commoncrawl.org/ccbot). Last checked: October 8, 2026

## AI bots: crawlers, tokens and controls

- **September 2022:** The IETF publishes the Robots Exclusion Protocol as RFC 9309 on the Standards Track. It defines how crawlers interpret the rules in a robots.txt file. ([Source](https://www.rfc-editor.org/rfc/rfc9309.html))
- **2023:** OpenAI introduces web crawler permissions for AI. Site owners can use them to state whether their content may be used for AI training. ([Source](https://openai.com/index/approach-to-data-and-ai/))
- **September 28, 2023:** Google introduces Google-Extended. This robots.txt token controls whether content helps improve Bard and the Vertex AI generative APIs. ([Source](https://blog.google/technology/ai/an-update-on-web-publisher-controls/))
- **February 8, 2024:** After Bard is renamed, Google's documentation lists Gemini Apps as a product affected by Google-Extended. ([Source](https://developers.google.com/search/updates#updated-the-description-of-the-google-extended-product-token_1))
- **June 10, 2024:** Apple introduces the foundation models behind Apple Intelligence. Publishers can opt out of training use, according to Apple's documentation via Applebot-Extended. ([Source](https://machinelearning.apple.com/research/introducing-apple-foundation-models))
- **July 3, 2024:** Cloudflare lets customers block AI bots, scrapers and crawlers with a single click. ([Source](https://blog.cloudflare.com/declaring-your-aindependence-block-ai-bots-scrapers-and-crawlers-with-a-single-click/))
- **July 25, 2024:** With the SearchGPT prototype, OpenAI separates search from training. Sites can appear in search results even if they opt out of AI training. ([Source](https://openai.com/index/searchgpt-prototype/))
- **August 20, 2024:** Google adds Google-CloudVertexBot to its list of crawlers. It crawls sites at the owner's request to build Vertex AI Agents. ([Source](https://developers.google.com/search/updates#introducing-the-google-cloudvertexbot-crawler))
- **April 25, 2025:** Based on publisher feedback, Google clarifies the description of the Google-Extended token. ([Source](https://developers.google.com/search/updates#updated-the-description-of-the-google-extended-product-token))
- **May 15, 2025:** Cloudflare proposes that bots and agents cryptographically sign their requests, based on HTTP Message Signatures (RFC 9421). ([Source](https://blog.cloudflare.com/web-bot-auth/))
- **July 1, 2025:** Cloudflare changes its default to block AI crawlers unless site owners decide otherwise. ([Source](https://blog.cloudflare.com/content-independence-day-no-ai-crawl-without-compensation/))
- **July 1, 2025:** Cloudflare launches pay per crawl as a private beta. Crawlers either pay per request or receive a 402 status code with a price. ([Source](https://blog.cloudflare.com/introducing-pay-per-crawl/))

## Not to be confused with

- [GPTBot](https://seo-geo-glossary.com/en/glossary/gptbot/index.md): collects content that may be used to train OpenAI's foundation models. According to OpenAI, blocking GPTBot does not affect ChatGPT search.

## Related terms

- [GPTBot](https://seo-geo-glossary.com/en/glossary/gptbot/index.md)
- [ChatGPT-User](https://seo-geo-glossary.com/en/glossary/chatgpt-user/index.md)
- [Claude-SearchBot](https://seo-geo-glossary.com/en/glossary/claude-searchbot/index.md)
- [AI search crawler](https://seo-geo-glossary.com/en/glossary/ai-search-crawler/index.md)
- [AI crawler](https://seo-geo-glossary.com/en/glossary/ai-crawler/index.md)
- [robots.txt](https://seo-geo-glossary.com/en/glossary/robots-txt/index.md)
- [Crawler IP ranges](https://seo-geo-glossary.com/en/glossary/crawler-ip-ranges/index.md)
- [Bot verification](https://seo-geo-glossary.com/en/glossary/bot-verification/index.md)

## Sources

- [Overview of OpenAI crawlers – OpenAI (fetched 2026-10-06)](https://developers.openai.com/api/docs/bots)
- [OAI-SearchBot IP ranges (JSON) – OpenAI](https://openai.com/searchbot.json)

---

Last updated: October 8, 2026
URL: https://seo-geo-glossary.com/en/glossary/oai-searchbot/