Blocking AI bots means denying specific AI crawlers and fetchers access to a website. The usual tool is robots.txt, sometimes backed by rules on the server or at a CDN.
At a glance
- Type
- Measure (robots.txt rules, backed by server or CDN rules)
- Affects
- AI crawlers and user-triggered fetches
- Values
- One token per bot, e.g.
GPTBot,ClaudeBot,OAI-SearchBot
Last checked:
How does it work with robots.txt?
Each bot gets its own entry with its token, for example GPTBot, ClaudeBot or OAI-SearchBot. Operators split their bots by purpose. OpenAI states that OAI-SearchBot and GPTBot settings are independent, so a site can block training and still be eligible for ChatGPT search.
| Bot | Operator | Purpose | robots.txt token | robots.txt according to the operator | IP list published? |
|---|---|---|---|---|---|
| Google-Extended | Controls Gemini training and grounding (no crawler of its own) | Google-Extended | robots.txt token only; fetching is done by existing Google crawlers | n/a (no user agent of its own) | |
| GPTBot | OpenAI | Training | GPTBot | Disallowing it signals content should not be used to train foundation models | yes (openai.) |
| OAI-SearchBot | OpenAI | Search (ChatGPT) | OAI-SearchBot | opted-out sites are not shown in ChatGPT search answers | yes (openai.) |
| ChatGPT-User | OpenAI | User-triggered fetch | ChatGPT-User | robots.txt rules may not apply, says OpenAI (user-initiated) | yes (openai.) |
| ClaudeBot | Anthropic | Training | ClaudeBot | yes, Anthropic says its bots honor robots.txt | yes (claude.) |
| Claude-SearchBot | Anthropic | Search (Claude) | Claude-SearchBot | yes, Anthropic says its bots honor robots.txt | yes (claude.) |
| PerplexityBot | Perplexity | Search (Perplexity), not used for model training | PerplexityBot | controllable via robots.txt; the docs contain no explicit compliance statement | yes (perplexity.) |
| Applebot-Extended | Apple | Controls AI training (does not crawl, says Apple) | Applebot-Extended | robots.txt token only, applies to content Applebot already crawled | n/a (no crawler of its own) |
| CCBot | Common Crawl | Open web crawl archive | CCBot | can be blocked via robots.txt, says Common Crawl | yes (index.) |
| Meta-ExternalAgent | Meta | Training foundation AI models, improving products | meta-externalagent | can be blocked via robots.txt, says Meta | not stated by the operator |
A selection of ten AI bots from major operators, not a complete list. All values come from each operator's own documentation (as of 8 October 2026). The table does not show whether a bot actually complies with robots.txt; it only reflects what operators state. User agents can be spoofed; published IP lists or reverse DNS identify genuine requests. Sources: OpenAI: Overview of OpenAI Crawlers; Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler? (updated 2026-04-07); Perplexity: Perplexity Crawlers; Google: Google's common crawlers (last updated 2026-07-14); Google: Overview of Google crawlers and fetchers (last updated 2026-06-12); Google: Verifying Google crawlers and fetchers (IP range files); Apple: About Applebot (2026-09-04); Meta: Meta Web Crawlers; Common Crawl: CCBot. Last checked: .
Why do sites block AI bots?
Cost is a reason besides protecting content. In 2024, Read the Docs reported that a single AI crawler downloaded 73 TB in May, running up more than $5,000 in bandwidth charges. After the project blocked crawlers that Cloudflare identified as AI bots, bandwidth for downloaded files fell by 75%.
Where are the limits?
RFC 9309 describes robots.txt rules as not a form of access authorization. They only work for bots that follow them. For user-triggered fetches, behavior varies. OpenAI says rules may not apply to ChatGPT-User, Google says its user-triggered fetchers generally ignore robots.txt, and Perplexity says the same about Perplexity-User.
Blocking can cost visibility. A site that blocks OAI-SearchBot will not appear in ChatGPT search answers, according to OpenAI.
Example
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
Both training crawlers are blocked. Search bots and user-triggered fetchers can still reach the site.
Checklist
- Read each bot’s purpose in the operator’s documentation (training, search or user-triggered fetch). OpenAI states that the GPTBot and OAI-SearchBot settings are independent of each other.
- Copy the robots.txt token of the bot exactly from the operator’s documentation.
- Block a search bot only if the site should stay out of the AI search answers. According to OpenAI, sites that opt out of OAI-SearchBot do not appear in ChatGPT search answers.
- Place robots.txt at the root of the domain (
/robots.txt), as RFC 9309 specifies. - Allow for the delay after a change. OpenAI mentions about 24 hours for OAI-SearchBot, DuckDuckGo 72 hours for DuckAssistBot.
- Do not rely on robots.txt alone for user-triggered fetches, because OpenAI, Google and Perplexity name exceptions there.
- Compare requests in the server log with the operators’ published IP lists. IP addresses can be personal data (CJEU, judgment of 19 October 2016, C-582/14, Breyer). This is not legal advice.
FAQ
Does robots.txt protect content? Only against bots that honor it. Access control needs rules on the server or at the CDN.
More on the searchVIU blog
Sources
- RFC 9309: Robots Exclusion Protocol – IETFrfc-editor.org
- Overview of OpenAI Crawlers – OpenAIdevelopers.openai.com
- Google User-Triggered Fetchers – Google Search Centraldevelopers.google.com
- Perplexity Crawlers – Perplexity Docsdocs.perplexity.ai
- The Bot Dilemma – When Bot Protection Becomes Customer Rejection – searchVIU blog (2025)searchviu.com
- AI crawlers need to be more respectful – Read the Docs Blog (2024)about.readthedocs.com
- Urteil C-582/14 Breyer – EuGHeur-lex.europa.eu
Last updated: