A training crawler is an AI crawler that fetches web pages to collect content for training AI models. The term is a common shorthand and not a category the operators define themselves. It separates these bots from crawlers that feed an AI search and from fetchers that open a page because a user asked.
At a glance
- Type
- Umbrella term for crawlers (not an official category)
- Affects
- training data for AI models
- Values
GPTBot,ClaudeBot; at Google the tokenGoogle-Extended
Last checked:
Which training crawlers exist?
OpenAI describes GPTBot as crawling content that may be used in training its generative AI foundation models. Disallowing it signals that a site’s content should not be used for that. Anthropic describes ClaudeBot as collecting web content that could potentially contribute to training its models.
Google works differently. Google-Extended is a robots.txt token with no separate user agent. It controls whether content crawled by Google can be used for training Gemini models and for grounding, and Google says it does not affect inclusion in Google Search.
| Bot | Operator | Purpose | robots.txt token | robots.txt according to the operator | IP list published? |
|---|---|---|---|---|---|
| Google-Extended | Controls Gemini training and grounding (no crawler of its own) | Google-Extended | robots.txt token only; fetching is done by existing Google crawlers | n/a (no user agent of its own) | |
| GPTBot | OpenAI | Training | GPTBot | Disallowing it signals content should not be used to train foundation models | yes (openai.) |
| OAI-SearchBot | OpenAI | Search (ChatGPT) | OAI-SearchBot | opted-out sites are not shown in ChatGPT search answers | yes (openai.) |
| ChatGPT-User | OpenAI | User-triggered fetch | ChatGPT-User | robots.txt rules may not apply, says OpenAI (user-initiated) | yes (openai.) |
| ClaudeBot | Anthropic | Training | ClaudeBot | yes, Anthropic says its bots honor robots.txt | yes (claude.) |
| Claude-SearchBot | Anthropic | Search (Claude) | Claude-SearchBot | yes, Anthropic says its bots honor robots.txt | yes (claude.) |
| PerplexityBot | Perplexity | Search (Perplexity), not used for model training | PerplexityBot | controllable via robots.txt; the docs contain no explicit compliance statement | yes (perplexity.) |
| Applebot-Extended | Apple | Controls AI training (does not crawl, says Apple) | Applebot-Extended | robots.txt token only, applies to content Applebot already crawled | n/a (no crawler of its own) |
| CCBot | Common Crawl | Open web crawl archive | CCBot | can be blocked via robots.txt, says Common Crawl | yes (index.) |
| Meta-ExternalAgent | Meta | Training foundation AI models, improving products | meta-externalagent | can be blocked via robots.txt, says Meta | not stated by the operator |
A selection of ten AI bots from major operators, not a complete list. All values come from each operator's own documentation (as of 8 October 2026). The table does not show whether a bot actually complies with robots.txt; it only reflects what operators state. User agents can be spoofed; published IP lists or reverse DNS identify genuine requests. Sources: OpenAI: Overview of OpenAI Crawlers; Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler? (updated 2026-04-07); Perplexity: Perplexity Crawlers; Google: Google's common crawlers (last updated 2026-07-14); Google: Overview of Google crawlers and fetchers (last updated 2026-06-12); Google: Verifying Google crawlers and fetchers (IP range files); Apple: About Applebot (2026-09-04); Meta: Meta Web Crawlers; Common Crawl: CCBot. Last checked: .
How do you control them?
Through robots.txt, using each bot’s token. OpenAI and Anthropic run separate bots for search and user requests, so training can be blocked on its own.
Example
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
Both bots are blocked from the whole site. The operators’ search and user bots are not affected by these rules.
Sources
- Overview of OpenAI Crawlers – OpenAIdevelopers.openai.com
- Does Anthropic crawl data from the web? – Claude Help Centersupport.claude.com
- Google Common Crawlers (Google-Extended) – Google Search Centraldevelopers.google.com
Last updated: