# Claude-User

Category: [AI Bots & Crawlers](https://seo-geo-glossary.com/en/glossary/category/ai-bots-crawlers/) › [OpenAI, Anthropic & Perplexity bots](https://seo-geo-glossary.com/en/glossary/category/ai-bots-crawlers/openai-anthropic-perplexity/)

**Claude-User** is the user agent Claude, Anthropic's AI assistant, uses to access websites when a person asks Claude a question. The visit is triggered by that request and is not broad crawling. The robots.txt token is `Claude-User`.

## What does Anthropic use Claude-User for?

According to Anthropic, Claude uses this user agent to visit websites in response to user queries. Anthropic runs two more bots with separate tasks: ClaudeBot collects web content for training, and Claude-SearchBot improves the quality of search results. A visit from Claude-User therefore happens at a specific moment when someone asked a matching question.

## How can you control Claude-User?

Anthropic states that its bots honor robots.txt directives, and that Claude-User lets site owners control which sites can be accessed through user-initiated requests. Unlike OpenAI for ChatGPT-User, Anthropic names no exception for user-triggered fetches. The non-standard Crawl-delay extension is supported as well. Blocking Claude-User, according to Anthropic, may reduce a site's visibility for user-directed web search. Requests can be checked against the IP ranges Anthropic publishes as a JSON file.

## Example

```robots
User-agent: Claude-User
Disallow: /private/
```

The rule blocks only the `/private/` directory. The rest of the site stays available to Claude-User.

## Common mistakes

- **Blocking Anthropic bots by IP address.** Anthropic warns that this stops the bot from reading robots.txt and may not guarantee an opt-out.
- **Setting the rule on the main domain only.** Anthropic says the directive must be applied to each subdomain separately.

## FAQ

**Does blocking Claude-User affect training?**
Training collection is the job of ClaudeBot. Each Anthropic bot has its own user agent and can be controlled separately.

**How does Claude-User differ from ChatGPT-User?**
Both fetch pages when a user asks a question. One belongs to Anthropic, the other to OpenAI. OpenAI notes that robots.txt rules may not apply to ChatGPT-User, while Anthropic documents robots.txt control for Claude-User.

## At a glance

- **Operator:** Anthropic
- **Purpose:** fetching pages when users ask Claude a question
- **robots.txt token:** Claude-User
- **Respects robots.txt:** yes, blocking prevents retrieval for user queries, says Anthropic
- **IP list:** [yes (JSON)](https://claude.com/crawling/bots.json)
- **Docs:** [Claude Help Center: Does Anthropic crawl data from the web](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)

Last checked: October 8, 2026

## AI bots compared

| Bot | Operator | Purpose | robots.txt token | robots.txt according to the operator | IP list published? |
|---|---|---|---|---|---|
| Google-Extended | Google | Controls Gemini training and grounding (no crawler of its own) | Google-Extended | robots.txt token only; fetching is done by existing Google crawlers | n/a (no user agent of its own) |
| GPTBot | OpenAI | Training | GPTBot | Disallowing it signals content should not be used to train foundation models | yes (openai.com/gptbot.json) |
| OAI-SearchBot | OpenAI | Search (ChatGPT) | OAI-SearchBot | opted-out sites are not shown in ChatGPT search answers | yes (openai.com/searchbot.json) |
| ChatGPT-User | OpenAI | User-triggered fetch | ChatGPT-User | robots.txt rules may not apply, says OpenAI (user-initiated) | yes (openai.com/chatgpt-user.json) |
| ClaudeBot | Anthropic | Training | ClaudeBot | yes, Anthropic says its bots honor robots.txt | yes (claude.com/crawling/bots.json) |
| Claude-SearchBot | Anthropic | Search (Claude) | Claude-SearchBot | yes, Anthropic says its bots honor robots.txt | yes (claude.com/crawling/bots.json) |
| PerplexityBot | Perplexity | Search (Perplexity), not used for model training | PerplexityBot | controllable via robots.txt; the docs contain no explicit compliance statement | yes (perplexity.com/perplexitybot.json) |
| Applebot-Extended | Apple | Controls AI training (does not crawl, says Apple) | Applebot-Extended | robots.txt token only, applies to content Applebot already crawled | n/a (no crawler of its own) |
| CCBot | Common Crawl | Open web crawl archive | CCBot | can be blocked via robots.txt, says Common Crawl | yes (index.commoncrawl.org/ccbot.json) |
| Meta-ExternalAgent | Meta | Training foundation AI models, improving products | meta-externalagent | can be blocked via robots.txt, says Meta | not stated by the operator |

Sources: [OpenAI: Overview of OpenAI Crawlers](https://developers.openai.com/api/docs/bots); [Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler? (updated 2026-04-07)](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler); [Perplexity: Perplexity Crawlers](https://docs.perplexity.ai/guides/bots); [Google: Google's common crawlers (last updated 2026-07-14)](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers); [Google: Overview of Google crawlers and fetchers (last updated 2026-06-12)](https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers); [Google: Verifying Google crawlers and fetchers (IP range files)](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests); [Apple: About Applebot (2026-09-04)](https://support.apple.com/en-us/119829); [Meta: Meta Web Crawlers](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/); [Common Crawl: CCBot](https://commoncrawl.org/ccbot). Last checked: October 8, 2026

## Not to be confused with

- [ClaudeBot](https://seo-geo-glossary.com/en/glossary/claudebot/index.md): collects web content that, according to Anthropic, may contribute to training its generative models. It has its own user agent and runs independently of user requests.

## Related terms

- [ChatGPT-User](https://seo-geo-glossary.com/en/glossary/chatgpt-user/index.md)
- [Claude-SearchBot](https://seo-geo-glossary.com/en/glossary/claude-searchbot/index.md)
- [Perplexity-User](https://seo-geo-glossary.com/en/glossary/perplexity-user/index.md)
- [ClaudeBot](https://seo-geo-glossary.com/en/glossary/claudebot/index.md)

## More on the searchVIU blog

- [What the Claude Code leak reveals about AI grounding — and what it means for your GEO strategy](https://www.searchviu.com/en/claude-leak-ai-grounding/)

## Sources

- [Does Anthropic crawl data from the web, and how can site owners block the crawler? – Claude Help Center](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)

---

Last updated: October 8, 2026
URL: https://seo-geo-glossary.com/en/glossary/claude-user/