# anthropic-ai

Category: [AI Bots & Crawlers](https://seo-geo-glossary.com/en/glossary/category/ai-bots-crawlers/) › [OpenAI, Anthropic & Perplexity bots](https://seo-geo-glossary.com/en/glossary/category/ai-bots-crawlers/openai-anthropic-perplexity/) · Historical

**anthropic-ai** is a user agent name that appears in many robots.txt files to keep Anthropic's crawlers out. Anthropic's current crawler documentation does not mention it. What counts today are the three bots listed there, `ClaudeBot`, `Claude-User` and `Claude-SearchBot`.

## What is documented about anthropic-ai?

Anthropic's help article on web crawling names three bots and explains the purpose of each and how to control it in robots.txt. It does not contain a token called `anthropic-ai`. According to reports, for example in third-party bot directories, `anthropic-ai` is an older name that Anthropic no longer documents. Anthropic has not confirmed this.

For robots.txt this has a clear consequence. Under RFC 9309, a rule group applies to the crawler whose user agent token matches the group. A rule for a name no bot uses has no effect.

## Example

This robots.txt blocks the three documented Anthropic bots. The `anthropic-ai` line can stay as an extra entry:

```robots
User-agent: ClaudeBot
User-agent: Claude-User
User-agent: Claude-SearchBot
User-agent: anthropic-ai
Disallow: /
```

To exclude only training, block `ClaudeBot` alone. The other two bots serve user requests and search.

## Common mistakes

- **Blocking only anthropic-ai.** Anthropic's help article names three bots with their own names. A rule for an undocumented name does not replace those entries.
- **Blocking Anthropic bots by IP address.** Anthropic warns that this stops the bot from reading robots.txt and may not guarantee an opt-out.
- **Setting the rule on the main domain only.** Anthropic says the directive must be applied to each subdomain separately.

## FAQ

**Does a rule for anthropic-ai block Anthropic?**
Not according to the current documentation. The three user agents Anthropic names are the ones that count.

**Does an outdated entry cause harm?**
It only has an effect if a bot uses that name. Keeping it next to the current tokens does no damage.

## At a glance

- **Operator:** Anthropic (attribution according to reports)
- **robots.txt token:** anthropic-ai
- **Purpose:** not described in Anthropic's documentation
- **Respects robots.txt:** not described in Anthropic's documentation
- **Status:** not listed in Anthropic's current crawler documentation
- **Docs:** [Anthropic help article: documents ClaudeBot, Claude-User and Claude-SearchBot](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)

Last checked: October 8, 2026

## AI bots compared

| Bot | Operator | Purpose | robots.txt token | robots.txt according to the operator | IP list published? |
|---|---|---|---|---|---|
| Google-Extended | Google | Controls Gemini training and grounding (no crawler of its own) | Google-Extended | robots.txt token only; fetching is done by existing Google crawlers | n/a (no user agent of its own) |
| GPTBot | OpenAI | Training | GPTBot | Disallowing it signals content should not be used to train foundation models | yes (openai.com/gptbot.json) |
| OAI-SearchBot | OpenAI | Search (ChatGPT) | OAI-SearchBot | opted-out sites are not shown in ChatGPT search answers | yes (openai.com/searchbot.json) |
| ChatGPT-User | OpenAI | User-triggered fetch | ChatGPT-User | robots.txt rules may not apply, says OpenAI (user-initiated) | yes (openai.com/chatgpt-user.json) |
| ClaudeBot | Anthropic | Training | ClaudeBot | yes, Anthropic says its bots honor robots.txt | yes (claude.com/crawling/bots.json) |
| Claude-SearchBot | Anthropic | Search (Claude) | Claude-SearchBot | yes, Anthropic says its bots honor robots.txt | yes (claude.com/crawling/bots.json) |
| PerplexityBot | Perplexity | Search (Perplexity), not used for model training | PerplexityBot | controllable via robots.txt; the docs contain no explicit compliance statement | yes (perplexity.com/perplexitybot.json) |
| Applebot-Extended | Apple | Controls AI training (does not crawl, says Apple) | Applebot-Extended | robots.txt token only, applies to content Applebot already crawled | n/a (no crawler of its own) |
| CCBot | Common Crawl | Open web crawl archive | CCBot | can be blocked via robots.txt, says Common Crawl | yes (index.commoncrawl.org/ccbot.json) |
| Meta-ExternalAgent | Meta | Training foundation AI models, improving products | meta-externalagent | can be blocked via robots.txt, says Meta | not stated by the operator |

Sources: [OpenAI: Overview of OpenAI Crawlers](https://developers.openai.com/api/docs/bots); [Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler? (updated 2026-04-07)](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler); [Perplexity: Perplexity Crawlers](https://docs.perplexity.ai/guides/bots); [Google: Google's common crawlers (last updated 2026-07-14)](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers); [Google: Overview of Google crawlers and fetchers (last updated 2026-06-12)](https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers); [Google: Verifying Google crawlers and fetchers (IP range files)](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests); [Apple: About Applebot (2026-09-04)](https://support.apple.com/en-us/119829); [Meta: Meta Web Crawlers](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/); [Common Crawl: CCBot](https://commoncrawl.org/ccbot). Last checked: October 8, 2026

## Related terms

- [ClaudeBot](https://seo-geo-glossary.com/en/glossary/claudebot/index.md)
- [Claude-User](https://seo-geo-glossary.com/en/glossary/claude-user/index.md)
- [Claude-SearchBot](https://seo-geo-glossary.com/en/glossary/claude-searchbot/index.md)
- [robots.txt](https://seo-geo-glossary.com/en/glossary/robots-txt/index.md)
- [User agent](https://seo-geo-glossary.com/en/glossary/user-agent/index.md)

## Sources

- [Does Anthropic crawl data from the web, and how can site owners block the crawler? – Claude Help Center](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)
- [RFC 9309: Robots Exclusion Protocol – IETF](https://www.rfc-editor.org/rfc/rfc9309.html)

---

Last updated: October 8, 2026
URL: https://seo-geo-glossary.com/en/glossary/anthropic-ai/