# Training crawler

Category: [AI Bots & Crawlers](https://seo-geo-glossary.com/en/glossary/category/ai-bots-crawlers/) › [Types of AI bots](https://seo-geo-glossary.com/en/glossary/category/ai-bots-crawlers/types-of-ai-bots/)

A **training crawler** is an AI crawler that fetches web pages to collect content for training AI models. The term is a common shorthand and not a category the operators define themselves. It separates these bots from crawlers that feed an AI search and from fetchers that open a page because a user asked.

## Which training crawlers exist?

OpenAI describes GPTBot as crawling content that may be used in training its generative AI foundation models. Disallowing it signals that a site's content should not be used for that. Anthropic describes ClaudeBot as collecting web content that could potentially contribute to training its models.

Google works differently. Google-Extended is a robots.txt token with no separate user agent. It controls whether content crawled by Google can be used for training Gemini models and for grounding, and Google says it does not affect inclusion in Google Search.

## How do you control them?

Through robots.txt, using each bot's token. OpenAI and Anthropic run separate bots for search and user requests, so training can be blocked on its own.

## Example

```robots
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /
```

Both bots are blocked from the whole site. The operators' search and user bots are not affected by these rules.

## At a glance

- **Type:** Umbrella term for crawlers (not an official category)
- **Affects:** training data for AI models
- **Values:** GPTBot, ClaudeBot; at Google the token Google-Extended
- **Docs:** [OpenAI: Overview of OpenAI Crawlers](https://developers.openai.com/api/docs/bots)

Last checked: October 8, 2026

## AI bots compared

| Bot | Operator | Purpose | robots.txt token | robots.txt according to the operator | IP list published? |
|---|---|---|---|---|---|
| Google-Extended | Google | Controls Gemini training and grounding (no crawler of its own) | Google-Extended | robots.txt token only; fetching is done by existing Google crawlers | n/a (no user agent of its own) |
| GPTBot | OpenAI | Training | GPTBot | Disallowing it signals content should not be used to train foundation models | yes (openai.com/gptbot.json) |
| OAI-SearchBot | OpenAI | Search (ChatGPT) | OAI-SearchBot | opted-out sites are not shown in ChatGPT search answers | yes (openai.com/searchbot.json) |
| ChatGPT-User | OpenAI | User-triggered fetch | ChatGPT-User | robots.txt rules may not apply, says OpenAI (user-initiated) | yes (openai.com/chatgpt-user.json) |
| ClaudeBot | Anthropic | Training | ClaudeBot | yes, Anthropic says its bots honor robots.txt | yes (claude.com/crawling/bots.json) |
| Claude-SearchBot | Anthropic | Search (Claude) | Claude-SearchBot | yes, Anthropic says its bots honor robots.txt | yes (claude.com/crawling/bots.json) |
| PerplexityBot | Perplexity | Search (Perplexity), not used for model training | PerplexityBot | controllable via robots.txt; the docs contain no explicit compliance statement | yes (perplexity.com/perplexitybot.json) |
| Applebot-Extended | Apple | Controls AI training (does not crawl, says Apple) | Applebot-Extended | robots.txt token only, applies to content Applebot already crawled | n/a (no crawler of its own) |
| CCBot | Common Crawl | Open web crawl archive | CCBot | can be blocked via robots.txt, says Common Crawl | yes (index.commoncrawl.org/ccbot.json) |
| Meta-ExternalAgent | Meta | Training foundation AI models, improving products | meta-externalagent | can be blocked via robots.txt, says Meta | not stated by the operator |

Sources: [OpenAI: Overview of OpenAI Crawlers](https://developers.openai.com/api/docs/bots); [Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler? (updated 2026-04-07)](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler); [Perplexity: Perplexity Crawlers](https://docs.perplexity.ai/guides/bots); [Google: Google's common crawlers (last updated 2026-07-14)](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers); [Google: Overview of Google crawlers and fetchers (last updated 2026-06-12)](https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers); [Google: Verifying Google crawlers and fetchers (IP range files)](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests); [Apple: About Applebot (2026-09-04)](https://support.apple.com/en-us/119829); [Meta: Meta Web Crawlers](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/); [Common Crawl: CCBot](https://commoncrawl.org/ccbot). Last checked: October 8, 2026

## Related terms

- [AI crawler](https://seo-geo-glossary.com/en/glossary/ai-crawler/index.md)
- [AI search crawler](https://seo-geo-glossary.com/en/glossary/ai-search-crawler/index.md)
- [Web crawler](https://seo-geo-glossary.com/en/glossary/web-crawler/index.md)
- [GPTBot](https://seo-geo-glossary.com/en/glossary/gptbot/index.md)
- [Google-Extended](https://seo-geo-glossary.com/en/glossary/google-extended/index.md)
- [CCBot](https://seo-geo-glossary.com/en/glossary/ccbot/index.md)
- [robots.txt](https://seo-geo-glossary.com/en/glossary/robots-txt/index.md)

## Sources

- [Overview of OpenAI Crawlers – OpenAI](https://developers.openai.com/api/docs/bots)
- [Does Anthropic crawl data from the web? – Claude Help Center](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)
- [Google Common Crawlers (Google-Extended) – Google Search Central](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers)

---

Last updated: October 8, 2026
URL: https://seo-geo-glossary.com/en/glossary/training-crawler/