# AI search crawler

Category: [AI Bots & Crawlers](https://seo-geo-glossary.com/en/glossary/category/ai-bots-crawlers/) › [Types of AI bots](https://seo-geo-glossary.com/en/glossary/category/ai-bots-crawlers/types-of-ai-bots/)

An **AI search crawler** is a crawler that fetches web pages for the search feature of an AI product, so the product can surface and link them in answers. Operators say these bots are not used for model training. The name is shorthand and not an official category.

## Which AI search crawlers exist?

OpenAI says OAI-SearchBot is used to surface websites in search results in ChatGPT's search features. Perplexity says PerplexityBot is designed to surface and link websites in Perplexity search results and is not used to crawl content for AI foundation models. Anthropic describes Claude-SearchBot as navigating the web to improve search result quality for users.

OpenAI notes that sites opting out of OAI-SearchBot will not appear in ChatGPT search answers and that a robots.txt change can take about 24 hours to take effect.

## How does it differ from other AI bots?

Training crawlers collect content for models. User-triggered fetchers load one page because someone asked. According to their operators, search crawlers fetch pages so the service can cite and link them in search answers.

## Example

```robots
User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /
```

The site stays open to ChatGPT search and is closed to OpenAI's training crawler.

## At a glance

- **Type:** Umbrella term for crawlers (not an official category)
- **Affects:** search feature of an AI product
- **Values:** OAI-SearchBot, Claude-SearchBot, PerplexityBot
- **Docs:** [OpenAI: Overview of OpenAI Crawlers](https://developers.openai.com/api/docs/bots)

Last checked: October 8, 2026

## AI bots compared

| Bot | Operator | Purpose | robots.txt token | robots.txt according to the operator | IP list published? |
|---|---|---|---|---|---|
| Google-Extended | Google | Controls Gemini training and grounding (no crawler of its own) | Google-Extended | robots.txt token only; fetching is done by existing Google crawlers | n/a (no user agent of its own) |
| GPTBot | OpenAI | Training | GPTBot | Disallowing it signals content should not be used to train foundation models | yes (openai.com/gptbot.json) |
| OAI-SearchBot | OpenAI | Search (ChatGPT) | OAI-SearchBot | opted-out sites are not shown in ChatGPT search answers | yes (openai.com/searchbot.json) |
| ChatGPT-User | OpenAI | User-triggered fetch | ChatGPT-User | robots.txt rules may not apply, says OpenAI (user-initiated) | yes (openai.com/chatgpt-user.json) |
| ClaudeBot | Anthropic | Training | ClaudeBot | yes, Anthropic says its bots honor robots.txt | yes (claude.com/crawling/bots.json) |
| Claude-SearchBot | Anthropic | Search (Claude) | Claude-SearchBot | yes, Anthropic says its bots honor robots.txt | yes (claude.com/crawling/bots.json) |
| PerplexityBot | Perplexity | Search (Perplexity), not used for model training | PerplexityBot | controllable via robots.txt; the docs contain no explicit compliance statement | yes (perplexity.com/perplexitybot.json) |
| Applebot-Extended | Apple | Controls AI training (does not crawl, says Apple) | Applebot-Extended | robots.txt token only, applies to content Applebot already crawled | n/a (no crawler of its own) |
| CCBot | Common Crawl | Open web crawl archive | CCBot | can be blocked via robots.txt, says Common Crawl | yes (index.commoncrawl.org/ccbot.json) |
| Meta-ExternalAgent | Meta | Training foundation AI models, improving products | meta-externalagent | can be blocked via robots.txt, says Meta | not stated by the operator |

Sources: [OpenAI: Overview of OpenAI Crawlers](https://developers.openai.com/api/docs/bots); [Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler? (updated 2026-04-07)](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler); [Perplexity: Perplexity Crawlers](https://docs.perplexity.ai/guides/bots); [Google: Google's common crawlers (last updated 2026-07-14)](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers); [Google: Overview of Google crawlers and fetchers (last updated 2026-06-12)](https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers); [Google: Verifying Google crawlers and fetchers (IP range files)](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests); [Apple: About Applebot (2026-09-04)](https://support.apple.com/en-us/119829); [Meta: Meta Web Crawlers](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/); [Common Crawl: CCBot](https://commoncrawl.org/ccbot). Last checked: October 8, 2026

## Related terms

- [AI crawler](https://seo-geo-glossary.com/en/glossary/ai-crawler/index.md)
- [Training crawler](https://seo-geo-glossary.com/en/glossary/training-crawler/index.md)
- [User-triggered fetcher](https://seo-geo-glossary.com/en/glossary/user-triggered-fetcher/index.md)
- [robots.txt](https://seo-geo-glossary.com/en/glossary/robots-txt/index.md)

## Sources

- [Overview of OpenAI Crawlers – OpenAI](https://developers.openai.com/api/docs/bots)
- [Perplexity Crawlers – Perplexity Docs](https://docs.perplexity.ai/docs/resources/perplexity-crawlers)
- [Does Anthropic crawl data from the web? – Claude Help Center](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)

---

Last updated: October 8, 2026
URL: https://seo-geo-glossary.com/en/glossary/ai-search-crawler/