# Meta-ExternalAgent Category: [AI Bots & Crawlers](https://seo-geo-glossary.com/en/glossary/category/ai-bots-crawlers/) › [Other AI bots](https://seo-geo-glossary.com/en/glossary/category/ai-bots-crawlers/other-ai-bots/) **Meta-ExternalAgent** is a web crawler run by Meta, the company behind Facebook, Instagram and WhatsApp. According to Meta, it crawls the web for use cases such as training AI foundation models or improving products by indexing content directly. It can be blocked with robots.txt rules. ## How does it differ from Meta-WebIndexer? Meta describes a second crawler, **Meta-WebIndexer**, on the same documentation page. It crawls to improve the quality of Meta AI search results. Meta suggests allowing it so that Meta AI can cite and link to your content. Both crawlers have their own user agent names and can be controlled separately. ## Example This entry blocks the crawler for the whole site: ```robots User-agent: meta-externalagent Disallow: / ``` To block only part of a site, use that path instead of `/`, for example `/internal/`. ## What site owners should check robots.txt is a request that a crawler has to honor. Server log files show whether and how fast requests stop after a change, since the crawler appears there as `meta-externalagent/1.1`, or in a variant with a link to Meta's documentation appended. Site owners have reported heavy request volumes from Meta crawlers in forums, but these reports are anecdotal. ## Common mistakes - **Expecting changes right away.** Meta asks site owners to allow up to 24 hours for robots.txt changes to take effect. - **Controlling only one Meta crawler.** Meta lists Meta-ExternalAgent, Meta-WebIndexer and Meta-ExternalFetcher under their own user agent names. An entry for `meta-externalagent` does not name the other two. ## FAQ **Is Meta-ExternalAgent used only for AI training?** Meta names AI model training and product improvement as example uses. The documentation does not limit it to one purpose. ## At a glance - **Operator:** Meta Platforms - **Purpose:** training AI foundation models, improving products (per Meta) - **robots.txt token:** meta-externalagent - **User agent:** meta-externalagent/1.1 - **Respects robots.txt:** can be blocked via robots.txt; changes can take up to 24 hours, says Meta - **Docs:** [Meta Web Crawlers](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/) Last checked: October 8, 2026 ## AI bots compared | Bot | Operator | Purpose | robots.txt token | robots.txt according to the operator | IP list published? | |---|---|---|---|---|---| | Google-Extended | Google | Controls Gemini training and grounding (no crawler of its own) | Google-Extended | robots.txt token only; fetching is done by existing Google crawlers | n/a (no user agent of its own) | | GPTBot | OpenAI | Training | GPTBot | Disallowing it signals content should not be used to train foundation models | yes (openai.com/gptbot.json) | | OAI-SearchBot | OpenAI | Search (ChatGPT) | OAI-SearchBot | opted-out sites are not shown in ChatGPT search answers | yes (openai.com/searchbot.json) | | ChatGPT-User | OpenAI | User-triggered fetch | ChatGPT-User | robots.txt rules may not apply, says OpenAI (user-initiated) | yes (openai.com/chatgpt-user.json) | | ClaudeBot | Anthropic | Training | ClaudeBot | yes, Anthropic says its bots honor robots.txt | yes (claude.com/crawling/bots.json) | | Claude-SearchBot | Anthropic | Search (Claude) | Claude-SearchBot | yes, Anthropic says its bots honor robots.txt | yes (claude.com/crawling/bots.json) | | PerplexityBot | Perplexity | Search (Perplexity), not used for model training | PerplexityBot | controllable via robots.txt; the docs contain no explicit compliance statement | yes (perplexity.com/perplexitybot.json) | | Applebot-Extended | Apple | Controls AI training (does not crawl, says Apple) | Applebot-Extended | robots.txt token only, applies to content Applebot already crawled | n/a (no crawler of its own) | | CCBot | Common Crawl | Open web crawl archive | CCBot | can be blocked via robots.txt, says Common Crawl | yes (index.commoncrawl.org/ccbot.json) | | Meta-ExternalAgent | Meta | Training foundation AI models, improving products | meta-externalagent | can be blocked via robots.txt, says Meta | not stated by the operator | Sources: [OpenAI: Overview of OpenAI Crawlers](https://developers.openai.com/api/docs/bots); [Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler? (updated 2026-04-07)](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler); [Perplexity: Perplexity Crawlers](https://docs.perplexity.ai/guides/bots); [Google: Google's common crawlers (last updated 2026-07-14)](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers); [Google: Overview of Google crawlers and fetchers (last updated 2026-06-12)](https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers); [Google: Verifying Google crawlers and fetchers (IP range files)](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests); [Apple: About Applebot (2026-09-04)](https://support.apple.com/en-us/119829); [Meta: Meta Web Crawlers](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/); [Common Crawl: CCBot](https://commoncrawl.org/ccbot). Last checked: October 8, 2026 ## Related terms - [robots.txt](https://seo-geo-glossary.com/en/glossary/robots-txt/index.md) - [User agent](https://seo-geo-glossary.com/en/glossary/user-agent/index.md) - [Blocking AI bots](https://seo-geo-glossary.com/en/glossary/blocking-ai-bots/index.md) - [Training crawler](https://seo-geo-glossary.com/en/glossary/training-crawler/index.md) - [Meta-WebIndexer](https://seo-geo-glossary.com/en/glossary/meta-webindexer/index.md) - [Meta-ExternalFetcher](https://seo-geo-glossary.com/en/glossary/meta-externalfetcher/index.md) ## Sources - [Meta Web Crawlers – Meta for Developers](https://developers.facebook.com/documentation/sharing/webmasters/web-crawlers) --- Last updated: October 8, 2026 URL: https://seo-geo-glossary.com/en/glossary/meta-externalagent/