Meta-ExternalFetcher is a Meta user agent that fetches individual links when a user asks for them. Meta says it supports agentic AI features, such as an AI navigating a website to complete a task for someone, and states that it may bypass robots.txt rules.
At a glance
- Operator
- Meta Platforms
- Purpose
- fetches individual links at a user's request (per Meta)
- robots.txt token
meta-externalfetcher- User agent
meta-externalfetcher/1.1- Respects robots.txt
- may bypass robots.txt, says Meta
Last checked:
How is it different from other Meta bots?
Meta lists its crawlers on a single documentation page. Meta-ExternalAgent crawls for training foundation models and for indexing content, and Meta-WebIndexer crawls to improve search results in Meta AI. Both run systematically and can be controlled in robots.txt. The ExternalFetcher only loads a page at the moment a person’s request needs it, which makes it a user-triggered fetcher similar to those other AI providers run.
As a result, a Disallow rule keeps Meta’s training and search crawlers out, while Meta does not commit to the ExternalFetcher following it. Blocking those requests reliably takes a rule at the server or CDN level (assessment).
| Bot | Operator | Purpose | robots.txt token | robots.txt according to the operator | IP list published? |
|---|---|---|---|---|---|
| Google-Extended | Controls Gemini training and grounding (no crawler of its own) | Google-Extended | robots.txt token only; fetching is done by existing Google crawlers | n/a (no user agent of its own) | |
| GPTBot | OpenAI | Training | GPTBot | Disallowing it signals content should not be used to train foundation models | yes (openai.) |
| OAI-SearchBot | OpenAI | Search (ChatGPT) | OAI-SearchBot | opted-out sites are not shown in ChatGPT search answers | yes (openai.) |
| ChatGPT-User | OpenAI | User-triggered fetch | ChatGPT-User | robots.txt rules may not apply, says OpenAI (user-initiated) | yes (openai.) |
| ClaudeBot | Anthropic | Training | ClaudeBot | yes, Anthropic says its bots honor robots.txt | yes (claude.) |
| Claude-SearchBot | Anthropic | Search (Claude) | Claude-SearchBot | yes, Anthropic says its bots honor robots.txt | yes (claude.) |
| PerplexityBot | Perplexity | Search (Perplexity), not used for model training | PerplexityBot | controllable via robots.txt; the docs contain no explicit compliance statement | yes (perplexity.) |
| Applebot-Extended | Apple | Controls AI training (does not crawl, says Apple) | Applebot-Extended | robots.txt token only, applies to content Applebot already crawled | n/a (no crawler of its own) |
| CCBot | Common Crawl | Open web crawl archive | CCBot | can be blocked via robots.txt, says Common Crawl | yes (index.) |
| Meta-ExternalAgent | Meta | Training foundation AI models, improving products | meta-externalagent | can be blocked via robots.txt, says Meta | not stated by the operator |
A selection of ten AI bots from major operators, not a complete list. All values come from each operator's own documentation (as of 8 October 2026). The table does not show whether a bot actually complies with robots.txt; it only reflects what operators state. User agents can be spoofed; published IP lists or reverse DNS identify genuine requests. Sources: OpenAI: Overview of OpenAI Crawlers; Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler? (updated 2026-04-07); Perplexity: Perplexity Crawlers; Google: Google's common crawlers (last updated 2026-07-14); Google: Overview of Google crawlers and fetchers (last updated 2026-06-12); Google: Verifying Google crawlers and fetchers (IP range files); Apple: About Applebot (2026-09-04); Meta: Meta Web Crawlers; Common Crawl: CCBot. Last checked: .
Example
A request in the access log carries this user agent:
meta-externalfetcher/1.1
Meta also lists a longer variant that adds a link to its crawler documentation. Some third-party bot directories describe the fetcher as robots.txt-compliant or as verifiable through signed requests. Meta’s documentation says the opposite about robots.txt and does not mention signing. A single hit on a product page shortly after someone asked Meta AI about that product fits this fetcher. Large, systematic crawls of many URLs point to one of the crawlers instead (assessment).
Common mistakes
- Treating
Disallowas a block. Meta writes that the fetcher may bypass robots.txt because it performs fetches requested by a user. - Expecting changes right away. For robots.txt changes, Meta’s documentation allows up to 24 hours.
FAQ
Does Meta-ExternalFetcher respect robots.txt? Not necessarily. Meta says it may bypass robots.txt rules because it acts on a user’s request.
Is it used to train AI models? Meta does not describe it that way. Model training is the stated purpose of Meta-ExternalAgent.
Sources
- Meta Web Crawlers – Meta for Developers (Meta-ExternalFetcher section, checked 2026-10-07)developers.facebook.com
Last updated: