# AI crawler

Category: [AI Bots & Crawlers](https://seo-geo-glossary.com/en/glossary/category/ai-bots-crawlers/) › [Types of AI bots](https://seo-geo-glossary.com/en/glossary/category/ai-bots-crawlers/types-of-ai-bots/) · Also known as: AI bot, AI web crawler, LLM crawler

An **AI crawler** is an automated program that fetches web pages for AI applications. Companies such as OpenAI, Anthropic, and Perplexity use them to collect training data for language models, to build search indexes for AI answers, or to open a page on behalf of a user in a chat.

## What types of AI crawlers are there?

The large AI companies now separate their bots by purpose. Each has its own user agent and can be controlled on its own in robots.txt.

- **Training crawlers** collect content that may be used to train future models, for example GPTBot (OpenAI) and ClaudeBot (Anthropic).
- **Search crawlers** build an index that AI answers with source links draw from, for example OAI-SearchBot, Claude-SearchBot, and PerplexityBot.
- **User-triggered fetchers** load a page because a person asked for it in a chat, for example ChatGPT-User, Claude-User, and Perplexity-User.

OpenAI states that each of its settings is independent, so a site can allow OAI-SearchBot and block GPTBot. For ChatGPT-User, OpenAI notes that robots.txt rules may not apply because a user starts the request. Perplexity says that Perplexity-User generally ignores robots.txt for the same reason. Cloudflare uses a similar split in its traffic data (training, search, user action) and adds a fourth group for bots with undeclared purposes.

Google handles this differently. **Google-Extended** is a robots.txt product token with no user agent of its own. Crawling happens with the existing Google crawlers, and the token controls whether that content may be used to train Gemini models and to ground their answers. Google states that Google-Extended does not affect inclusion in Google Search and is not a ranking signal.

## Example

A site wants to appear in ChatGPT search results but keep its content out of model training:

```robots
User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /
```

## What should site owners watch for?

robots.txt asks crawlers to stay out of certain areas. It does not technically block anyone. Anthropic says its bots honor robots.txt. OpenAI documents separate tokens for GPTBot and OAI-SearchBot that sites use in robots.txt to control how their content is used. Perplexity recommends allowing PerplexityBot in robots.txt. Any client can send any user agent, though. To confirm that a request really comes from a vendor, compare its IP address with the ranges the vendor publishes as JSON files. Which AI crawlers actually visit a site, how often, and which pages they request is visible only in the server log files.

## Common mistakes

- **Treating training and search the same.** OpenAI says the settings for its bots are independent. Sites that block OAI-SearchBot are not shown in ChatGPT search answers, even if GPTBot is allowed.
- **Expecting user-triggered fetchers to follow robots.txt.** OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them.
- **Blocking Googlebot to prevent AI training.** That is what the Google-Extended token is for. Blocking Googlebot removes the site from Google Search.
- **Blocking by IP address instead of robots.txt.** Anthropic notes that its bots then cannot read robots.txt, so the opt-out may not work reliably.

## FAQ

**Is ChatGPT a web crawler?**
ChatGPT itself is a chat assistant. OpenAI runs several separate bots around it: GPTBot for training, OAI-SearchBot for search, and ChatGPT-User for fetches a user starts.

**Does blocking AI crawlers hurt Google rankings?**
Blocking Google-Extended does not affect Google Search, according to Google. Blocking Googlebot removes a site from Google Search, and with it from AI Overviews and AI Mode.

## At a glance

- **Operator:** AI companies, e.g. OpenAI, Anthropic, Perplexity, Google
- **Purpose:** Training, search, or fetching on behalf of a user
- **robots.txt token:** one token per bot, e.g. GPTBot, ClaudeBot, Google-Extended
- **Respects robots.txt:** depends on the operator; user-triggered fetchers may not follow it
- **IP list:** published as JSON by many operators, e.g. openai.com/gptbot.json
- **Docs:** [Operator docs, e.g. OpenAI: Overview of OpenAI Crawlers](https://developers.openai.com/api/docs/bots)

Last checked: October 8, 2026

## AI bots compared

| Bot | Operator | Purpose | robots.txt token | robots.txt according to the operator | IP list published? |
|---|---|---|---|---|---|
| Google-Extended | Google | Controls Gemini training and grounding (no crawler of its own) | Google-Extended | robots.txt token only; fetching is done by existing Google crawlers | n/a (no user agent of its own) |
| GPTBot | OpenAI | Training | GPTBot | Disallowing it signals content should not be used to train foundation models | yes (openai.com/gptbot.json) |
| OAI-SearchBot | OpenAI | Search (ChatGPT) | OAI-SearchBot | opted-out sites are not shown in ChatGPT search answers | yes (openai.com/searchbot.json) |
| ChatGPT-User | OpenAI | User-triggered fetch | ChatGPT-User | robots.txt rules may not apply, says OpenAI (user-initiated) | yes (openai.com/chatgpt-user.json) |
| ClaudeBot | Anthropic | Training | ClaudeBot | yes, Anthropic says its bots honor robots.txt | yes (claude.com/crawling/bots.json) |
| Claude-SearchBot | Anthropic | Search (Claude) | Claude-SearchBot | yes, Anthropic says its bots honor robots.txt | yes (claude.com/crawling/bots.json) |
| PerplexityBot | Perplexity | Search (Perplexity), not used for model training | PerplexityBot | controllable via robots.txt; the docs contain no explicit compliance statement | yes (perplexity.com/perplexitybot.json) |
| Applebot-Extended | Apple | Controls AI training (does not crawl, says Apple) | Applebot-Extended | robots.txt token only, applies to content Applebot already crawled | n/a (no crawler of its own) |
| CCBot | Common Crawl | Open web crawl archive | CCBot | can be blocked via robots.txt, says Common Crawl | yes (index.commoncrawl.org/ccbot.json) |
| Meta-ExternalAgent | Meta | Training foundation AI models, improving products | meta-externalagent | can be blocked via robots.txt, says Meta | not stated by the operator |

Sources: [OpenAI: Overview of OpenAI Crawlers](https://developers.openai.com/api/docs/bots); [Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler? (updated 2026-04-07)](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler); [Perplexity: Perplexity Crawlers](https://docs.perplexity.ai/guides/bots); [Google: Google's common crawlers (last updated 2026-07-14)](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers); [Google: Overview of Google crawlers and fetchers (last updated 2026-06-12)](https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers); [Google: Verifying Google crawlers and fetchers (IP range files)](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests); [Apple: About Applebot (2026-09-04)](https://support.apple.com/en-us/119829); [Meta: Meta Web Crawlers](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/); [Common Crawl: CCBot](https://commoncrawl.org/ccbot). Last checked: October 8, 2026

## AI bots: crawlers, tokens and controls

- **September 2022:** The IETF publishes the Robots Exclusion Protocol as RFC 9309 on the Standards Track. It defines how crawlers interpret the rules in a robots.txt file. ([Source](https://www.rfc-editor.org/rfc/rfc9309.html))
- **2023:** OpenAI introduces web crawler permissions for AI. Site owners can use them to state whether their content may be used for AI training. ([Source](https://openai.com/index/approach-to-data-and-ai/))
- **September 28, 2023:** Google introduces Google-Extended. This robots.txt token controls whether content helps improve Bard and the Vertex AI generative APIs. ([Source](https://blog.google/technology/ai/an-update-on-web-publisher-controls/))
- **February 8, 2024:** After Bard is renamed, Google's documentation lists Gemini Apps as a product affected by Google-Extended. ([Source](https://developers.google.com/search/updates#updated-the-description-of-the-google-extended-product-token_1))
- **June 10, 2024:** Apple introduces the foundation models behind Apple Intelligence. Publishers can opt out of training use, according to Apple's documentation via Applebot-Extended. ([Source](https://machinelearning.apple.com/research/introducing-apple-foundation-models))
- **July 3, 2024:** Cloudflare lets customers block AI bots, scrapers and crawlers with a single click. ([Source](https://blog.cloudflare.com/declaring-your-aindependence-block-ai-bots-scrapers-and-crawlers-with-a-single-click/))
- **July 25, 2024:** With the SearchGPT prototype, OpenAI separates search from training. Sites can appear in search results even if they opt out of AI training. ([Source](https://openai.com/index/searchgpt-prototype/))
- **August 20, 2024:** Google adds Google-CloudVertexBot to its list of crawlers. It crawls sites at the owner's request to build Vertex AI Agents. ([Source](https://developers.google.com/search/updates#introducing-the-google-cloudvertexbot-crawler))
- **April 25, 2025:** Based on publisher feedback, Google clarifies the description of the Google-Extended token. ([Source](https://developers.google.com/search/updates#updated-the-description-of-the-google-extended-product-token))
- **May 15, 2025:** Cloudflare proposes that bots and agents cryptographically sign their requests, based on HTTP Message Signatures (RFC 9421). ([Source](https://blog.cloudflare.com/web-bot-auth/))
- **July 1, 2025:** Cloudflare changes its default to block AI crawlers unless site owners decide otherwise. ([Source](https://blog.cloudflare.com/content-independence-day-no-ai-crawl-without-compensation/))
- **July 1, 2025:** Cloudflare launches pay per crawl as a private beta. Crawlers either pay per request or receive a 402 status code with a price. ([Source](https://blog.cloudflare.com/introducing-pay-per-crawl/))

## Related terms

- [Training crawler](https://seo-geo-glossary.com/en/glossary/training-crawler/index.md)
- [AI search crawler](https://seo-geo-glossary.com/en/glossary/ai-search-crawler/index.md)
- [User-triggered fetcher](https://seo-geo-glossary.com/en/glossary/user-triggered-fetcher/index.md)
- [Web crawler](https://seo-geo-glossary.com/en/glossary/web-crawler/index.md)
- [GPTBot](https://seo-geo-glossary.com/en/glossary/gptbot/index.md)
- [ClaudeBot](https://seo-geo-glossary.com/en/glossary/claudebot/index.md)
- [Google-Extended](https://seo-geo-glossary.com/en/glossary/google-extended/index.md)
- [robots.txt](https://seo-geo-glossary.com/en/glossary/robots-txt/index.md)
- [Bot verification](https://seo-geo-glossary.com/en/glossary/bot-verification/index.md)
- [Log file analysis](https://seo-geo-glossary.com/en/glossary/log-file-analysis/index.md)

## More on the searchVIU blog

- [AI Crawlers & JavaScript Rendering](https://www.searchviu.com/en/ai-crawlers-javascript-rendering/)

## Sources

- [Overview of OpenAI crawlers – OpenAI](https://developers.openai.com/api/docs/bots)
- [Does Anthropic crawl data from the web, and how can site owners block the crawler? – Claude Help Center (updated 2026-04-07)](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)
- [Perplexity crawlers – Perplexity Docs](https://docs.perplexity.ai/guides/bots)
- [Google's common crawlers (Google-Extended) – Google Search Central](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers)
- [AI features and your website – Google Search Central (last updated 2025-12-10)](https://developers.google.com/search/docs/appearance/ai-features)
- [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309)
- [A deeper look at AI crawlers: breaking down traffic by purpose and industry – Cloudflare Blog, 2025-08-28](https://blog.cloudflare.com/ai-crawler-traffic-by-purpose-and-industry/)

---

Last updated: October 8, 2026
URL: https://seo-geo-glossary.com/en/glossary/ai-crawler/