Meta-WebIndexer is a Meta crawler that, according to Meta, navigates the web to improve the quality of search results in Meta AI. Meta says that allowing it in robots.txt helps Meta cite and link to a site’s content in Meta AI answers.
At a glance
- Operator
- Meta Platforms
- Purpose
- improving Meta AI search result quality (per Meta)
- robots.txt token
meta-webindexer- User agent
meta-webindexer/1.1- Respects robots.txt
- controlled via robots.txt; changes can take up to 24 hours, says Meta
Last checked:
How is it different from Meta-ExternalAgent?
Meta documents several crawlers on one page. Meta-ExternalAgent is used for training foundation AI models or for indexing content directly for products. Meta-WebIndexer serves search in Meta AI. Each has its own user agent token, so robots.txt can treat them differently. Meta notes that robots.txt changes can take up to 24 hours to apply because of caching.
| Bot | Operator | Purpose | robots.txt token | robots.txt according to the operator | IP list published? |
|---|---|---|---|---|---|
| Google-Extended | Controls Gemini training and grounding (no crawler of its own) | Google-Extended | robots.txt token only; fetching is done by existing Google crawlers | n/a (no user agent of its own) | |
| GPTBot | OpenAI | Training | GPTBot | Disallowing it signals content should not be used to train foundation models | yes (openai.) |
| OAI-SearchBot | OpenAI | Search (ChatGPT) | OAI-SearchBot | opted-out sites are not shown in ChatGPT search answers | yes (openai.) |
| ChatGPT-User | OpenAI | User-triggered fetch | ChatGPT-User | robots.txt rules may not apply, says OpenAI (user-initiated) | yes (openai.) |
| ClaudeBot | Anthropic | Training | ClaudeBot | yes, Anthropic says its bots honor robots.txt | yes (claude.) |
| Claude-SearchBot | Anthropic | Search (Claude) | Claude-SearchBot | yes, Anthropic says its bots honor robots.txt | yes (claude.) |
| PerplexityBot | Perplexity | Search (Perplexity), not used for model training | PerplexityBot | controllable via robots.txt; the docs contain no explicit compliance statement | yes (perplexity.) |
| Applebot-Extended | Apple | Controls AI training (does not crawl, says Apple) | Applebot-Extended | robots.txt token only, applies to content Applebot already crawled | n/a (no crawler of its own) |
| CCBot | Common Crawl | Open web crawl archive | CCBot | can be blocked via robots.txt, says Common Crawl | yes (index.) |
| Meta-ExternalAgent | Meta | Training foundation AI models, improving products | meta-externalagent | can be blocked via robots.txt, says Meta | not stated by the operator |
A selection of ten AI bots from major operators, not a complete list. All values come from each operator's own documentation (as of 8 October 2026). The table does not show whether a bot actually complies with robots.txt; it only reflects what operators state. User agents can be spoofed; published IP lists or reverse DNS identify genuine requests. Sources: OpenAI: Overview of OpenAI Crawlers; Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler? (updated 2026-04-07); Perplexity: Perplexity Crawlers; Google: Google's common crawlers (last updated 2026-07-14); Google: Overview of Google crawlers and fetchers (last updated 2026-06-12); Google: Verifying Google crawlers and fetchers (IP range files); Apple: About Applebot (2026-09-04); Meta: Meta Web Crawlers; Common Crawl: CCBot. Last checked: .
Example
User-agent: meta-webindexer
Allow: /
User-agent: meta-externalagent
Disallow: /
This keeps a site available to Meta AI search while excluding the training crawler. The user agent contains the token meta-webindexer/1.1.
Common mistakes
- Blocking all Meta crawlers together. A Disallow for the search crawler excludes it. According to Meta, allowing it helps Meta AI cite and link to your content.
- Expecting changes right away. Meta allows up to 24 hours for an updated robots.txt to take effect.
FAQ
Does blocking meta-externalagent also block Meta-WebIndexer?
No. Meta lists separate tokens. A rule for one crawler does not apply to the other unless both are named or a User-agent: * group covers them.
Is Meta-WebIndexer used for AI training? Meta describes it as the crawler for Meta AI search results. Training of foundation models is the stated purpose of Meta-ExternalAgent.
Sources
Last updated: