An AI crawler is an automated program that fetches web pages for AI applications. Companies such as OpenAI, Anthropic, and Perplexity use them to collect training data for language models, to build search indexes for AI answers, or to open a page on behalf of a user in a chat.
At a glance
- Operator
- AI companies, e.g. OpenAI, Anthropic, Perplexity, Google
- Purpose
- Training, search, or fetching on behalf of a user
- robots.txt token
- one token per bot, e.g.
GPTBot,ClaudeBot,Google-Extended - Respects robots.txt
- depends on the operator; user-triggered fetchers may not follow it
- IP list
- published as JSON by many operators, e.g.
openai.com/gptbot.json
Last checked:
What types of AI crawlers are there?
The large AI companies now separate their bots by purpose. Each has its own user agent and can be controlled on its own in robots.txt.
- Training crawlers collect content that may be used to train future models, for example GPTBot (OpenAI) and ClaudeBot (Anthropic).
- Search crawlers build an index that AI answers with source links draw from, for example OAI-SearchBot, Claude-SearchBot, and PerplexityBot.
- User-triggered fetchers load a page because a person asked for it in a chat, for example ChatGPT-User, Claude-User, and Perplexity-User.
OpenAI states that each of its settings is independent, so a site can allow OAI-SearchBot and block GPTBot. For ChatGPT-User, OpenAI notes that robots.txt rules may not apply because a user starts the request. Perplexity says that Perplexity-User generally ignores robots.txt for the same reason. Cloudflare uses a similar split in its traffic data (training, search, user action) and adds a fourth group for bots with undeclared purposes.
Google handles this differently. Google-Extended is a robots.txt product token with no user agent of its own. Crawling happens with the existing Google crawlers, and the token controls whether that content may be used to train Gemini models and to ground their answers. Google states that Google-Extended does not affect inclusion in Google Search and is not a ranking signal.
| Bot | Operator | Purpose | robots.txt token | robots.txt according to the operator | IP list published? |
|---|---|---|---|---|---|
| Google-Extended | Controls Gemini training and grounding (no crawler of its own) | Google-Extended | robots.txt token only; fetching is done by existing Google crawlers | n/a (no user agent of its own) | |
| GPTBot | OpenAI | Training | GPTBot | Disallowing it signals content should not be used to train foundation models | yes (openai.) |
| OAI-SearchBot | OpenAI | Search (ChatGPT) | OAI-SearchBot | opted-out sites are not shown in ChatGPT search answers | yes (openai.) |
| ChatGPT-User | OpenAI | User-triggered fetch | ChatGPT-User | robots.txt rules may not apply, says OpenAI (user-initiated) | yes (openai.) |
| ClaudeBot | Anthropic | Training | ClaudeBot | yes, Anthropic says its bots honor robots.txt | yes (claude.) |
| Claude-SearchBot | Anthropic | Search (Claude) | Claude-SearchBot | yes, Anthropic says its bots honor robots.txt | yes (claude.) |
| PerplexityBot | Perplexity | Search (Perplexity), not used for model training | PerplexityBot | controllable via robots.txt; the docs contain no explicit compliance statement | yes (perplexity.) |
| Applebot-Extended | Apple | Controls AI training (does not crawl, says Apple) | Applebot-Extended | robots.txt token only, applies to content Applebot already crawled | n/a (no crawler of its own) |
| CCBot | Common Crawl | Open web crawl archive | CCBot | can be blocked via robots.txt, says Common Crawl | yes (index.) |
| Meta-ExternalAgent | Meta | Training foundation AI models, improving products | meta-externalagent | can be blocked via robots.txt, says Meta | not stated by the operator |
A selection of ten AI bots from major operators, not a complete list. All values come from each operator's own documentation (as of 8 October 2026). The table does not show whether a bot actually complies with robots.txt; it only reflects what operators state. User agents can be spoofed; published IP lists or reverse DNS identify genuine requests. Sources: OpenAI: Overview of OpenAI Crawlers; Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler? (updated 2026-04-07); Perplexity: Perplexity Crawlers; Google: Google's common crawlers (last updated 2026-07-14); Google: Overview of Google crawlers and fetchers (last updated 2026-06-12); Google: Verifying Google crawlers and fetchers (IP range files); Apple: About Applebot (2026-09-04); Meta: Meta Web Crawlers; Common Crawl: CCBot. Last checked: .
- The IETF publishes the Robots Exclusion Protocol as RFC 9309 on the Standards Track. It defines how crawlers interpret the rules in a robots.txt file.Source: rfc-editor.org
- OpenAI introduces web crawler permissions for AI. Site owners can use them to state whether their content may be used for AI training.Source: openai.com
- Google introduces Google-Extended. This robots.txt token controls whether content helps improve Bard and the Vertex AI generative APIs.Source: blog.google
- After Bard is renamed, Google's documentation lists Gemini Apps as a product affected by Google-Extended.Source: developers.google.com
- Apple introduces the foundation models behind Apple Intelligence. Publishers can opt out of training use, according to Apple's documentation via Applebot-Extended.Source: machinelearning.apple.com
- Cloudflare lets customers block AI bots, scrapers and crawlers with a single click.Source: blog.cloudflare.com
- With the SearchGPT prototype, OpenAI separates search from training. Sites can appear in search results even if they opt out of AI training.Source: openai.com
- Google adds Google-CloudVertexBot to its list of crawlers. It crawls sites at the owner's request to build Vertex AI Agents.Source: developers.google.com
- Based on publisher feedback, Google clarifies the description of the Google-Extended token.Source: developers.google.com
- Cloudflare proposes that bots and agents cryptographically sign their requests, based on HTTP Message Signatures (RFC 9421).Source: blog.cloudflare.com
- Cloudflare changes its default to block AI crawlers unless site owners decide otherwise.Source: blog.cloudflare.com
- Cloudflare launches pay per crawl as a private beta. Crawlers either pay per request or receive a 402 status code with a price.Source: blog.cloudflare.com
OpenAI only says “last summer” for its crawler permissions (post of May 7, 2024); the often-cited August 2023 date for GPTBot comes from press reports. ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, OAI-SearchBot, ChatGPT-User, Meta-ExternalAgent and CCBot are documented by their operators, but without a verifiable launch date. Last checked: .
Example
A site wants to appear in ChatGPT search results but keep its content out of model training:
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
What should site owners watch for?
robots.txt asks crawlers to stay out of certain areas. It does not technically block anyone. Anthropic says its bots honor robots.txt. OpenAI documents separate tokens for GPTBot and OAI-SearchBot that sites use in robots.txt to control how their content is used. Perplexity recommends allowing PerplexityBot in robots.txt. Any client can send any user agent, though. To confirm that a request really comes from a vendor, compare its IP address with the ranges the vendor publishes as JSON files. Which AI crawlers actually visit a site, how often, and which pages they request is visible only in the server log files.
Common mistakes
- Treating training and search the same. OpenAI says the settings for its bots are independent. Sites that block OAI-SearchBot are not shown in ChatGPT search answers, even if GPTBot is allowed.
- Expecting user-triggered fetchers to follow robots.txt. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them.
- Blocking Googlebot to prevent AI training. That is what the Google-Extended token is for. Blocking Googlebot removes the site from Google Search.
- Blocking by IP address instead of robots.txt. Anthropic notes that its bots then cannot read robots.txt, so the opt-out may not work reliably.
FAQ
Is ChatGPT a web crawler? ChatGPT itself is a chat assistant. OpenAI runs several separate bots around it: GPTBot for training, OAI-SearchBot for search, and ChatGPT-User for fetches a user starts.
Does blocking AI crawlers hurt Google rankings? Blocking Google-Extended does not affect Google Search, according to Google. Blocking Googlebot removes a site from Google Search, and with it from AI Overviews and AI Mode.
More on the searchVIU blog
Sources
- Overview of OpenAI crawlers – OpenAIdevelopers.openai.com
- Does Anthropic crawl data from the web, and how can site owners block the crawler? – Claude Help Center (updated 2026-04-07)support.claude.com
- Perplexity crawlers – Perplexity Docsdocs.perplexity.ai
- Google's common crawlers (Google-Extended) – Google Search Centraldevelopers.google.com
- AI features and your website – Google Search Central (last updated 2025-12-10)developers.google.com
- RFC 9309: Robots Exclusion Protocolrfc-editor.org
- A deeper look at AI crawlers: breaking down traffic by purpose and industry – Cloudflare Blog, 2025-08-28blog.cloudflare.com
Last updated: