# Blocking AI bots

Category: [AI Bots & Crawlers](https://seo-geo-glossary.com/en/glossary/category/ai-bots-crawlers/) › [Managing & measuring AI bots](https://seo-geo-glossary.com/en/glossary/category/ai-bots-crawlers/managing-measuring-ai-bots/)

**Blocking AI bots** means denying specific AI crawlers and fetchers access to a website. The usual tool is robots.txt, sometimes backed by rules on the server or at a CDN.

## How does it work with robots.txt?

Each bot gets its own entry with its token, for example `GPTBot`, `ClaudeBot` or `OAI-SearchBot`. Operators split their bots by purpose. OpenAI states that OAI-SearchBot and GPTBot settings are independent, so a site can block training and still be eligible for ChatGPT search.

## Why do sites block AI bots?

Cost is a reason besides protecting content. In 2024, Read the Docs reported that a single AI crawler downloaded 73 TB in May, running up more than $5,000 in bandwidth charges. After the project blocked crawlers that Cloudflare identified as AI bots, bandwidth for downloaded files fell by 75%.

## Where are the limits?

RFC 9309 describes robots.txt rules as not a form of access authorization. They only work for bots that follow them. For user-triggered fetches, behavior varies. OpenAI says rules may not apply to ChatGPT-User, Google says its user-triggered fetchers generally ignore robots.txt, and Perplexity says the same about Perplexity-User.

Blocking can cost visibility. A site that blocks OAI-SearchBot will not appear in ChatGPT search answers, according to OpenAI.

## Example

```robots
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /
```

Both training crawlers are blocked. Search bots and user-triggered fetchers can still reach the site.

## Checklist

- Read each bot's purpose in the operator's documentation (training, search or user-triggered fetch). OpenAI states that the GPTBot and OAI-SearchBot settings are independent of each other.
- Copy the robots.txt token of the bot exactly from the operator's documentation.
- Block a search bot only if the site should stay out of the AI search answers. According to OpenAI, sites that opt out of OAI-SearchBot do not appear in ChatGPT search answers.
- Place robots.txt at the root of the domain (`/robots.txt`), as RFC 9309 specifies.
- Allow for the delay after a change. OpenAI mentions about 24 hours for OAI-SearchBot, DuckDuckGo 72 hours for DuckAssistBot.
- Do not rely on robots.txt alone for user-triggered fetches, because OpenAI, Google and Perplexity name exceptions there.
- Compare requests in the server log with the operators' published IP lists. IP addresses can be personal data (CJEU, judgment of 19 October 2016, C-582/14, Breyer). This is not legal advice.

## FAQ

**Does robots.txt protect content?**
Only against bots that honor it. Access control needs rules on the server or at the CDN.

## At a glance

- **Type:** Measure (robots.txt rules, backed by server or CDN rules)
- **Affects:** AI crawlers and user-triggered fetches
- **Values:** One token per bot, e.g. GPTBot, ClaudeBot, OAI-SearchBot
- **Docs:** [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309)

Last checked: October 8, 2026

## AI bots compared

| Bot | Operator | Purpose | robots.txt token | robots.txt according to the operator | IP list published? |
|---|---|---|---|---|---|
| Google-Extended | Google | Controls Gemini training and grounding (no crawler of its own) | Google-Extended | robots.txt token only; fetching is done by existing Google crawlers | n/a (no user agent of its own) |
| GPTBot | OpenAI | Training | GPTBot | Disallowing it signals content should not be used to train foundation models | yes (openai.com/gptbot.json) |
| OAI-SearchBot | OpenAI | Search (ChatGPT) | OAI-SearchBot | opted-out sites are not shown in ChatGPT search answers | yes (openai.com/searchbot.json) |
| ChatGPT-User | OpenAI | User-triggered fetch | ChatGPT-User | robots.txt rules may not apply, says OpenAI (user-initiated) | yes (openai.com/chatgpt-user.json) |
| ClaudeBot | Anthropic | Training | ClaudeBot | yes, Anthropic says its bots honor robots.txt | yes (claude.com/crawling/bots.json) |
| Claude-SearchBot | Anthropic | Search (Claude) | Claude-SearchBot | yes, Anthropic says its bots honor robots.txt | yes (claude.com/crawling/bots.json) |
| PerplexityBot | Perplexity | Search (Perplexity), not used for model training | PerplexityBot | controllable via robots.txt; the docs contain no explicit compliance statement | yes (perplexity.com/perplexitybot.json) |
| Applebot-Extended | Apple | Controls AI training (does not crawl, says Apple) | Applebot-Extended | robots.txt token only, applies to content Applebot already crawled | n/a (no crawler of its own) |
| CCBot | Common Crawl | Open web crawl archive | CCBot | can be blocked via robots.txt, says Common Crawl | yes (index.commoncrawl.org/ccbot.json) |
| Meta-ExternalAgent | Meta | Training foundation AI models, improving products | meta-externalagent | can be blocked via robots.txt, says Meta | not stated by the operator |

Sources: [OpenAI: Overview of OpenAI Crawlers](https://developers.openai.com/api/docs/bots); [Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler? (updated 2026-04-07)](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler); [Perplexity: Perplexity Crawlers](https://docs.perplexity.ai/guides/bots); [Google: Google's common crawlers (last updated 2026-07-14)](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers); [Google: Overview of Google crawlers and fetchers (last updated 2026-06-12)](https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers); [Google: Verifying Google crawlers and fetchers (IP range files)](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests); [Apple: About Applebot (2026-09-04)](https://support.apple.com/en-us/119829); [Meta: Meta Web Crawlers](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/); [Common Crawl: CCBot](https://commoncrawl.org/ccbot). Last checked: October 8, 2026

## Related terms

- [AI crawler](https://seo-geo-glossary.com/en/glossary/ai-crawler/index.md)
- [robots.txt](https://seo-geo-glossary.com/en/glossary/robots-txt/index.md)
- [Training crawler](https://seo-geo-glossary.com/en/glossary/training-crawler/index.md)
- [AI search crawler](https://seo-geo-glossary.com/en/glossary/ai-search-crawler/index.md)
- [User-triggered fetcher](https://seo-geo-glossary.com/en/glossary/user-triggered-fetcher/index.md)
- [Web Bot Auth](https://seo-geo-glossary.com/en/glossary/web-bot-auth/index.md)

## More on the searchVIU blog

- [The Bot Dilemma – When Bot Protection Becomes Customer Rejection](https://www.searchviu.com/en/the-bot-dilemma-when-bot-protection-becomes-customer-rejection/)
- [Cloudflare Fundamentals for SEO & GEO](https://www.searchviu.com/en/cloudflare-for-seo-and-geo-part-1/)

## Sources

- [RFC 9309: Robots Exclusion Protocol – IETF](https://www.rfc-editor.org/rfc/rfc9309)
- [Overview of OpenAI Crawlers – OpenAI](https://developers.openai.com/api/docs/bots)
- [Google User-Triggered Fetchers – Google Search Central](https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers)
- [Perplexity Crawlers – Perplexity Docs](https://docs.perplexity.ai/docs/resources/perplexity-crawlers)
- [The Bot Dilemma – When Bot Protection Becomes Customer Rejection – searchVIU blog (2025)](https://www.searchviu.com/en/the-bot-dilemma-when-bot-protection-becomes-customer-rejection/)
- [AI crawlers need to be more respectful – Read the Docs Blog (2024)](https://about.readthedocs.com/blog/2024/07/ai-crawlers-abuse/)
- [Urteil C-582/14 Breyer – EuGH](https://eur-lex.europa.eu/legal-content/DE/TXT/?uri=CELEX:62014CJ0582)

---

Last updated: October 8, 2026
URL: https://seo-geo-glossary.com/en/glossary/blocking-ai-bots/