A user-triggered fetcher is an automated program that loads a web page because a person asked a product to do so. Google defines user-triggered fetchers as being initiated by users to perform a fetching function within a Google product.
At a glance
- Type
- Type of fetch (triggered by users, single URL)
- Affects
- exactly the requested page
- Values
ChatGPT-User,Claude-User,Perplexity-User
Last checked:
How do user-triggered fetchers treat robots.txt?
It depends on the operator. Google states that because a user requested the fetch, these fetchers generally ignore robots.txt rules. OpenAI says robots.txt rules may not apply to ChatGPT-User because users initiate those actions. Perplexity says Perplexity-User generally ignores robots.txt for the same reason. Anthropic says its bots honor robots.txt directives and lists Claude-User among them.
A robots.txt rule is therefore not a dependable way to stop these requests. Restricting them takes rules on the server or at a CDN.
| Bot | Operator | Purpose | robots.txt token | robots.txt according to the operator | IP list published? |
|---|---|---|---|---|---|
| Google-Extended | Controls Gemini training and grounding (no crawler of its own) | Google-Extended | robots.txt token only; fetching is done by existing Google crawlers | n/a (no user agent of its own) | |
| GPTBot | OpenAI | Training | GPTBot | Disallowing it signals content should not be used to train foundation models | yes (openai.) |
| OAI-SearchBot | OpenAI | Search (ChatGPT) | OAI-SearchBot | opted-out sites are not shown in ChatGPT search answers | yes (openai.) |
| ChatGPT-User | OpenAI | User-triggered fetch | ChatGPT-User | robots.txt rules may not apply, says OpenAI (user-initiated) | yes (openai.) |
| ClaudeBot | Anthropic | Training | ClaudeBot | yes, Anthropic says its bots honor robots.txt | yes (claude.) |
| Claude-SearchBot | Anthropic | Search (Claude) | Claude-SearchBot | yes, Anthropic says its bots honor robots.txt | yes (claude.) |
| PerplexityBot | Perplexity | Search (Perplexity), not used for model training | PerplexityBot | controllable via robots.txt; the docs contain no explicit compliance statement | yes (perplexity.) |
| Applebot-Extended | Apple | Controls AI training (does not crawl, says Apple) | Applebot-Extended | robots.txt token only, applies to content Applebot already crawled | n/a (no crawler of its own) |
| CCBot | Common Crawl | Open web crawl archive | CCBot | can be blocked via robots.txt, says Common Crawl | yes (index.) |
| Meta-ExternalAgent | Meta | Training foundation AI models, improving products | meta-externalagent | can be blocked via robots.txt, says Meta | not stated by the operator |
A selection of ten AI bots from major operators, not a complete list. All values come from each operator's own documentation (as of 8 October 2026). The table does not show whether a bot actually complies with robots.txt; it only reflects what operators state. User agents can be spoofed; published IP lists or reverse DNS identify genuine requests. Sources: OpenAI: Overview of OpenAI Crawlers; Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler? (updated 2026-04-07); Perplexity: Perplexity Crawlers; Google: Google's common crawlers (last updated 2026-07-14); Google: Overview of Google crawlers and fetchers (last updated 2026-06-12); Google: Verifying Google crawlers and fetchers (IP range files); Apple: About Applebot (2026-09-04); Meta: Meta Web Crawlers; Common Crawl: CCBot. Last checked: .
Which examples exist?
Google lists fetchers such as Google Read Aloud, Google Site Verifier, Feedfetcher and Google-Agent. Among AI assistants, ChatGPT-User, Claude-User and Perplexity-User work this way.
Example
Someone asks an AI assistant to summarize a product page. The assistant requests the URL once. The server log shows a single request with the service’s user agent and not a series of crawl requests.
FAQ
Is a user-triggered fetcher a crawler? Google files them under a separate heading from its crawlers. Crawlers run on their own, fetchers act when a user asks.
Sources
- Google User-Triggered Fetchers – Google Search Centraldevelopers.google.com
- Overview of OpenAI Crawlers – OpenAIdevelopers.openai.com
- Perplexity Crawlers – Perplexity Docsdocs.perplexity.ai
- Does Anthropic crawl data from the web? – Claude Help Centersupport.claude.com
Last updated: