# ClaudeBot

Category: [AI Bots & Crawlers](https://seo-geo-glossary.com/en/glossary/category/ai-bots-crawlers/) › [OpenAI, Anthropic & Perplexity bots](https://seo-geo-glossary.com/en/glossary/category/ai-bots-crawlers/openai-anthropic-perplexity/)

**ClaudeBot** is the web crawler of Anthropic, the company behind the AI assistant Claude. It collects publicly available web content that Anthropic may use for training and developing its models. In server logs it appears with the user-agent token `ClaudeBot`.

ClaudeBot should not be confused with "Clawdbot", a separate AI assistant project with a similar name that dominates many search results for the term.

## What does Anthropic use ClaudeBot for?

Anthropic runs three bots with separate tasks. **ClaudeBot** gathers content for model training. **Claude-User** fetches a website when a Claude user asks a question that requires it. **Claude-SearchBot** crawls the web to improve the quality of search results for users.

The split matters for site owners. Blocking ClaudeBot tells Anthropic that the site's future content should be excluded from training data. Claude-User and Claude-SearchBot have their own user agents and need their own rules.

Cloudflare's traffic data illustrates the difference between training and referral. For the first week of August 2025, Cloudflare reported a crawl-to-refer ratio of nearly 50,000 to 1 for Anthropic, meaning many crawled pages for each visit referred back to a site. This is a snapshot of one week, not a current value.

## How do you control ClaudeBot?

Anthropic says it respects robots.txt directives and also supports the non-standard `Crawl-delay` extension to slow down crawling. The rules have to be set on every subdomain that should be covered.

To verify that a request really comes from Anthropic, check the source IP against the list Anthropic publishes at `claude.com/crawling/bots.json`. Anthropic advises against blocking its crawlers by IP address, because the bots then can't read robots.txt, and the opt-out may not work reliably.

## Example

Block ClaudeBot from the whole site while still allowing Claude-User:

```robots
User-agent: ClaudeBot
Disallow: /

User-agent: Claude-User
Allow: /
```

Slow ClaudeBot down instead of blocking it (delay in seconds):

```robots
User-agent: ClaudeBot
Crawl-delay: 1
```

## Common mistakes

- **Blocking only the main domain.** Anthropic says the rule has to be set in robots.txt on every subdomain that should be opted out.
- **Blocking ClaudeBot by IP address.** Anthropic notes that the bot then cannot read robots.txt, so the opt-out may not work reliably.
- **Treating ClaudeBot as all Anthropic bots.** A ClaudeBot block concerns training. Claude-User and Claude-SearchBot have their own tokens and need their own rules.
- **Trusting the user agent alone.** It can be faked. Matching the source IP against Anthropic's published list shows whether a request really comes from Anthropic.

## FAQ

**Should I block ClaudeBot?**
That depends on whether you want your content used to train AI models. A ClaudeBot block concerns training. Sites that want to be cited in Claude's answers should look at Claude-User and Claude-SearchBot separately instead of blocking all Anthropic bots at once.

**How do I find ClaudeBot in my log files?**
Look for the token `ClaudeBot` in the user agent. Since that value can be faked, match the source IP against Anthropic's published list as well.

## At a glance

- **Operator:** Anthropic
- **Purpose:** Training (content that could contribute to model training)
- **robots.txt token:** ClaudeBot
- **Respects robots.txt:** yes, according to Anthropic; also Crawl-delay
- **IP list:** [claude.com/crawling/bots.json](https://claude.com/crawling/bots.json)
- **Docs:** [Anthropic: Does Anthropic crawl data from the web …](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)

Last checked: October 8, 2026

## AI bots compared

| Bot | Operator | Purpose | robots.txt token | robots.txt according to the operator | IP list published? |
|---|---|---|---|---|---|
| Google-Extended | Google | Controls Gemini training and grounding (no crawler of its own) | Google-Extended | robots.txt token only; fetching is done by existing Google crawlers | n/a (no user agent of its own) |
| GPTBot | OpenAI | Training | GPTBot | Disallowing it signals content should not be used to train foundation models | yes (openai.com/gptbot.json) |
| OAI-SearchBot | OpenAI | Search (ChatGPT) | OAI-SearchBot | opted-out sites are not shown in ChatGPT search answers | yes (openai.com/searchbot.json) |
| ChatGPT-User | OpenAI | User-triggered fetch | ChatGPT-User | robots.txt rules may not apply, says OpenAI (user-initiated) | yes (openai.com/chatgpt-user.json) |
| ClaudeBot | Anthropic | Training | ClaudeBot | yes, Anthropic says its bots honor robots.txt | yes (claude.com/crawling/bots.json) |
| Claude-SearchBot | Anthropic | Search (Claude) | Claude-SearchBot | yes, Anthropic says its bots honor robots.txt | yes (claude.com/crawling/bots.json) |
| PerplexityBot | Perplexity | Search (Perplexity), not used for model training | PerplexityBot | controllable via robots.txt; the docs contain no explicit compliance statement | yes (perplexity.com/perplexitybot.json) |
| Applebot-Extended | Apple | Controls AI training (does not crawl, says Apple) | Applebot-Extended | robots.txt token only, applies to content Applebot already crawled | n/a (no crawler of its own) |
| CCBot | Common Crawl | Open web crawl archive | CCBot | can be blocked via robots.txt, says Common Crawl | yes (index.commoncrawl.org/ccbot.json) |
| Meta-ExternalAgent | Meta | Training foundation AI models, improving products | meta-externalagent | can be blocked via robots.txt, says Meta | not stated by the operator |

Sources: [OpenAI: Overview of OpenAI Crawlers](https://developers.openai.com/api/docs/bots); [Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler? (updated 2026-04-07)](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler); [Perplexity: Perplexity Crawlers](https://docs.perplexity.ai/guides/bots); [Google: Google's common crawlers (last updated 2026-07-14)](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers); [Google: Overview of Google crawlers and fetchers (last updated 2026-06-12)](https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers); [Google: Verifying Google crawlers and fetchers (IP range files)](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests); [Apple: About Applebot (2026-09-04)](https://support.apple.com/en-us/119829); [Meta: Meta Web Crawlers](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/); [Common Crawl: CCBot](https://commoncrawl.org/ccbot). Last checked: October 8, 2026

## AI bots: crawlers, tokens and controls

- **September 2022:** The IETF publishes the Robots Exclusion Protocol as RFC 9309 on the Standards Track. It defines how crawlers interpret the rules in a robots.txt file. ([Source](https://www.rfc-editor.org/rfc/rfc9309.html))
- **2023:** OpenAI introduces web crawler permissions for AI. Site owners can use them to state whether their content may be used for AI training. ([Source](https://openai.com/index/approach-to-data-and-ai/))
- **September 28, 2023:** Google introduces Google-Extended. This robots.txt token controls whether content helps improve Bard and the Vertex AI generative APIs. ([Source](https://blog.google/technology/ai/an-update-on-web-publisher-controls/))
- **February 8, 2024:** After Bard is renamed, Google's documentation lists Gemini Apps as a product affected by Google-Extended. ([Source](https://developers.google.com/search/updates#updated-the-description-of-the-google-extended-product-token_1))
- **June 10, 2024:** Apple introduces the foundation models behind Apple Intelligence. Publishers can opt out of training use, according to Apple's documentation via Applebot-Extended. ([Source](https://machinelearning.apple.com/research/introducing-apple-foundation-models))
- **July 3, 2024:** Cloudflare lets customers block AI bots, scrapers and crawlers with a single click. ([Source](https://blog.cloudflare.com/declaring-your-aindependence-block-ai-bots-scrapers-and-crawlers-with-a-single-click/))
- **July 25, 2024:** With the SearchGPT prototype, OpenAI separates search from training. Sites can appear in search results even if they opt out of AI training. ([Source](https://openai.com/index/searchgpt-prototype/))
- **August 20, 2024:** Google adds Google-CloudVertexBot to its list of crawlers. It crawls sites at the owner's request to build Vertex AI Agents. ([Source](https://developers.google.com/search/updates#introducing-the-google-cloudvertexbot-crawler))
- **April 25, 2025:** Based on publisher feedback, Google clarifies the description of the Google-Extended token. ([Source](https://developers.google.com/search/updates#updated-the-description-of-the-google-extended-product-token))
- **May 15, 2025:** Cloudflare proposes that bots and agents cryptographically sign their requests, based on HTTP Message Signatures (RFC 9421). ([Source](https://blog.cloudflare.com/web-bot-auth/))
- **July 1, 2025:** Cloudflare changes its default to block AI crawlers unless site owners decide otherwise. ([Source](https://blog.cloudflare.com/content-independence-day-no-ai-crawl-without-compensation/))
- **July 1, 2025:** Cloudflare launches pay per crawl as a private beta. Crawlers either pay per request or receive a 402 status code with a price. ([Source](https://blog.cloudflare.com/introducing-pay-per-crawl/))

## Not to be confused with

- [Claude-User](https://seo-geo-glossary.com/en/glossary/claude-user/index.md): fetches websites when users ask Claude a question that requires reading a page. According to Anthropic, blocking it stops Claude from retrieving content for those user requests.

## Related terms

- [Claude-User](https://seo-geo-glossary.com/en/glossary/claude-user/index.md)
- [Claude-SearchBot](https://seo-geo-glossary.com/en/glossary/claude-searchbot/index.md)
- [AI crawler](https://seo-geo-glossary.com/en/glossary/ai-crawler/index.md)
- [Training crawler](https://seo-geo-glossary.com/en/glossary/training-crawler/index.md)
- [robots.txt](https://seo-geo-glossary.com/en/glossary/robots-txt/index.md)
- [Crawl-delay](https://seo-geo-glossary.com/en/glossary/crawl-delay/index.md)
- [Crawler IP ranges](https://seo-geo-glossary.com/en/glossary/crawler-ip-ranges/index.md)
- [Bot verification](https://seo-geo-glossary.com/en/glossary/bot-verification/index.md)

## More on the searchVIU blog

- [AI Crawlers & JavaScript Rendering](https://www.searchviu.com/en/ai-crawlers-javascript-rendering/)

## Sources

- [Does Anthropic crawl data from the web, and how can site owners block the crawler? – Claude Help Center (updated 2026-04-07)](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)
- [IP ranges of Anthropic's crawlers (bots.json)](https://claude.com/crawling/bots.json)
- [A deeper look at AI crawlers: breaking down traffic by purpose and industry – Cloudflare Blog, 2025-08-28](https://blog.cloudflare.com/ai-crawler-traffic-by-purpose-and-industry/)

---

Last updated: October 8, 2026
URL: https://seo-geo-glossary.com/en/glossary/claudebot/