# Claude-SearchBot

Category: [AI Bots & Crawlers](https://seo-geo-glossary.com/en/glossary/category/ai-bots-crawlers/) › [OpenAI, Anthropic & Perplexity bots](https://seo-geo-glossary.com/en/glossary/category/ai-bots-crawlers/openai-anthropic-perplexity/)

**Claude-SearchBot** is an Anthropic crawler that navigates the web to improve the quality of search results for Claude users. It is one of three Anthropic bots. ClaudeBot collects content for model training, and Claude-User fetches pages when a person asks Claude a question.

## What happens if you block it?

Anthropic states that blocking Claude-SearchBot may reduce a site's visibility and the accuracy of search results in Claude. Anyone who only wants to keep content out of training would block ClaudeBot instead, since each bot is controlled by its own user agent. Some third-party bot directories also link Claude-SearchBot to AI training, which Anthropic's description does not mention.

## Example

The pattern comes from Anthropic's ClaudeBot example, applied here to Claude-SearchBot:

```robots
User-agent: Claude-SearchBot
Disallow: /
```

Anthropic also documents the `Crawl-delay` directive for slowing the bots down:

```robots
User-agent: Claude-SearchBot
Crawl-delay: 1
```

## A note on IP blocking

Anthropic advises against blocking its bots by IP address. A bot blocked that way cannot read the site's robots.txt and does not learn about the site owner's preferences. For checking genuine requests, Anthropic publishes the official IP list.

## Common mistakes

- **Blocking Claude-SearchBot to exclude training.** According to Anthropic, blocking it removes content from search indexing. ClaudeBot collects training data.
- **Setting the rule on the main domain only.** Anthropic says the directive must be applied to each subdomain separately.

## FAQ

**Is Claude-SearchBot the same as ClaudeBot?**
No. They are separate bots with separate user agents and purposes.

## At a glance

- **Operator:** Anthropic
- **Purpose:** improving search result quality in Claude (indexing web content)
- **robots.txt token:** Claude-SearchBot
- **Respects robots.txt:** yes, Anthropic says its bots honor robots.txt
- **IP list:** [yes (JSON)](https://claude.com/crawling/bots.json)
- **Docs:** [Claude Help Center: Does Anthropic crawl data from the web](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)

Last checked: October 8, 2026

## AI bots compared

| Bot | Operator | Purpose | robots.txt token | robots.txt according to the operator | IP list published? |
|---|---|---|---|---|---|
| Google-Extended | Google | Controls Gemini training and grounding (no crawler of its own) | Google-Extended | robots.txt token only; fetching is done by existing Google crawlers | n/a (no user agent of its own) |
| GPTBot | OpenAI | Training | GPTBot | Disallowing it signals content should not be used to train foundation models | yes (openai.com/gptbot.json) |
| OAI-SearchBot | OpenAI | Search (ChatGPT) | OAI-SearchBot | opted-out sites are not shown in ChatGPT search answers | yes (openai.com/searchbot.json) |
| ChatGPT-User | OpenAI | User-triggered fetch | ChatGPT-User | robots.txt rules may not apply, says OpenAI (user-initiated) | yes (openai.com/chatgpt-user.json) |
| ClaudeBot | Anthropic | Training | ClaudeBot | yes, Anthropic says its bots honor robots.txt | yes (claude.com/crawling/bots.json) |
| Claude-SearchBot | Anthropic | Search (Claude) | Claude-SearchBot | yes, Anthropic says its bots honor robots.txt | yes (claude.com/crawling/bots.json) |
| PerplexityBot | Perplexity | Search (Perplexity), not used for model training | PerplexityBot | controllable via robots.txt; the docs contain no explicit compliance statement | yes (perplexity.com/perplexitybot.json) |
| Applebot-Extended | Apple | Controls AI training (does not crawl, says Apple) | Applebot-Extended | robots.txt token only, applies to content Applebot already crawled | n/a (no crawler of its own) |
| CCBot | Common Crawl | Open web crawl archive | CCBot | can be blocked via robots.txt, says Common Crawl | yes (index.commoncrawl.org/ccbot.json) |
| Meta-ExternalAgent | Meta | Training foundation AI models, improving products | meta-externalagent | can be blocked via robots.txt, says Meta | not stated by the operator |

Sources: [OpenAI: Overview of OpenAI Crawlers](https://developers.openai.com/api/docs/bots); [Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler? (updated 2026-04-07)](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler); [Perplexity: Perplexity Crawlers](https://docs.perplexity.ai/guides/bots); [Google: Google's common crawlers (last updated 2026-07-14)](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers); [Google: Overview of Google crawlers and fetchers (last updated 2026-06-12)](https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers); [Google: Verifying Google crawlers and fetchers (IP range files)](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests); [Apple: About Applebot (2026-09-04)](https://support.apple.com/en-us/119829); [Meta: Meta Web Crawlers](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/); [Common Crawl: CCBot](https://commoncrawl.org/ccbot). Last checked: October 8, 2026

## Related terms

- [ClaudeBot](https://seo-geo-glossary.com/en/glossary/claudebot/index.md)
- [Claude-User](https://seo-geo-glossary.com/en/glossary/claude-user/index.md)
- [OAI-SearchBot](https://seo-geo-glossary.com/en/glossary/oai-searchbot/index.md)
- [robots.txt](https://seo-geo-glossary.com/en/glossary/robots-txt/index.md)

## Sources

- [Does Anthropic crawl data from the web, and how can site owners block the crawler? – Claude Help Center](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)
- [Claude-SearchBot – CrawlerCheck (secondary)](https://crawlercheck.com/directory/ai-bots/claude-searchbot)

---

Last updated: October 8, 2026
URL: https://seo-geo-glossary.com/en/glossary/claude-searchbot/