# GPTBot

Category: [AI Bots & Crawlers](https://seo-geo-glossary.com/en/glossary/category/ai-bots-crawlers/) › [OpenAI, Anthropic & Perplexity bots](https://seo-geo-glossary.com/en/glossary/category/ai-bots-crawlers/openai-anthropic-perplexity/)

**GPTBot** is OpenAI's web crawler for collecting publicly available content that may be used to train its generative AI models. Site owners can allow or block it in robots.txt with the user-agent token `GPTBot`. According to reports, it became publicly known in August 2023.

## What does GPTBot do?

OpenAI says GPTBot is used to make its generative AI foundation models more useful and safe. Disallowing GPTBot signals that a site's content should not be used for training those models.

OpenAI runs other bots for other jobs. **OAI-SearchBot** surfaces websites in ChatGPT's search features, and **ChatGPT-User** fetches pages when a user takes an action in ChatGPT or a custom GPT. OpenAI states that each setting is independent of the others. Blocking GPTBot therefore does not remove a site from ChatGPT search.

## How do you identify GPTBot?

As of October 2026, OpenAI documents this user-agent string:

```txt
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot
```

The version number changes over time, so filters should match the token `GPTBot`. Any client can fake a user agent. OpenAI therefore publishes GPTBot's IP ranges at `openai.com/gptbot.json`. Comparing the IP address in the server logs with that list shows whether a request really came from OpenAI.

## Example

Block GPTBot but keep ChatGPT search access:

```robots
User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /
```

Block only certain directories for GPTBot:

```robots
User-agent: GPTBot
Disallow: /account/
Disallow: /internal/
```

## Common mistakes

- **Confusing GPTBot with the search bot.** To appear in ChatGPT search, OAI-SearchBot needs access. The GPTBot rule only concerns training.
- **Filtering by user agent alone.** Spoofed requests then show up as GPTBot in reports or get blocked by mistake. The IP list separates real from fake requests.
- **Overlooking blanket blocks at the host or CDN.** Bot protection settings can block AI crawlers without anything appearing in robots.txt (assessment).

## FAQ

**What is a GPT bot?**
In the SEO context, GPTBot is OpenAI's training crawler. The term is also used loosely for chatbots built on GPT models, which is a different thing.

**Does blocking GPTBot remove my site from ChatGPT?**
Not according to OpenAI. ChatGPT search relies on OAI-SearchBot, and the settings for each bot are independent.

**Does GPTBot render JavaScript?**
OpenAI does not say. Content that only appears after JavaScript runs may not be captured, so important text should be in the HTML the server delivers.

## At a glance

- **Operator:** OpenAI
- **Purpose:** Training generative AI foundation models
- **robots.txt token:** GPTBot
- **Respects robots.txt:** Disallowing it signals content should not be used for training
- **IP list:** [openai.com/gptbot.json](https://openai.com/gptbot.json)
- **Docs:** [OpenAI: Overview of OpenAI Crawlers](https://developers.openai.com/api/docs/bots)

Last checked: October 8, 2026

## AI bots compared

| Bot | Operator | Purpose | robots.txt token | robots.txt according to the operator | IP list published? |
|---|---|---|---|---|---|
| Google-Extended | Google | Controls Gemini training and grounding (no crawler of its own) | Google-Extended | robots.txt token only; fetching is done by existing Google crawlers | n/a (no user agent of its own) |
| GPTBot | OpenAI | Training | GPTBot | Disallowing it signals content should not be used to train foundation models | yes (openai.com/gptbot.json) |
| OAI-SearchBot | OpenAI | Search (ChatGPT) | OAI-SearchBot | opted-out sites are not shown in ChatGPT search answers | yes (openai.com/searchbot.json) |
| ChatGPT-User | OpenAI | User-triggered fetch | ChatGPT-User | robots.txt rules may not apply, says OpenAI (user-initiated) | yes (openai.com/chatgpt-user.json) |
| ClaudeBot | Anthropic | Training | ClaudeBot | yes, Anthropic says its bots honor robots.txt | yes (claude.com/crawling/bots.json) |
| Claude-SearchBot | Anthropic | Search (Claude) | Claude-SearchBot | yes, Anthropic says its bots honor robots.txt | yes (claude.com/crawling/bots.json) |
| PerplexityBot | Perplexity | Search (Perplexity), not used for model training | PerplexityBot | controllable via robots.txt; the docs contain no explicit compliance statement | yes (perplexity.com/perplexitybot.json) |
| Applebot-Extended | Apple | Controls AI training (does not crawl, says Apple) | Applebot-Extended | robots.txt token only, applies to content Applebot already crawled | n/a (no crawler of its own) |
| CCBot | Common Crawl | Open web crawl archive | CCBot | can be blocked via robots.txt, says Common Crawl | yes (index.commoncrawl.org/ccbot.json) |
| Meta-ExternalAgent | Meta | Training foundation AI models, improving products | meta-externalagent | can be blocked via robots.txt, says Meta | not stated by the operator |

Sources: [OpenAI: Overview of OpenAI Crawlers](https://developers.openai.com/api/docs/bots); [Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler? (updated 2026-04-07)](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler); [Perplexity: Perplexity Crawlers](https://docs.perplexity.ai/guides/bots); [Google: Google's common crawlers (last updated 2026-07-14)](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers); [Google: Overview of Google crawlers and fetchers (last updated 2026-06-12)](https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers); [Google: Verifying Google crawlers and fetchers (IP range files)](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests); [Apple: About Applebot (2026-09-04)](https://support.apple.com/en-us/119829); [Meta: Meta Web Crawlers](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/); [Common Crawl: CCBot](https://commoncrawl.org/ccbot). Last checked: October 8, 2026

## AI bots: crawlers, tokens and controls

- **September 2022:** The IETF publishes the Robots Exclusion Protocol as RFC 9309 on the Standards Track. It defines how crawlers interpret the rules in a robots.txt file. ([Source](https://www.rfc-editor.org/rfc/rfc9309.html))
- **2023:** OpenAI introduces web crawler permissions for AI. Site owners can use them to state whether their content may be used for AI training. ([Source](https://openai.com/index/approach-to-data-and-ai/))
- **September 28, 2023:** Google introduces Google-Extended. This robots.txt token controls whether content helps improve Bard and the Vertex AI generative APIs. ([Source](https://blog.google/technology/ai/an-update-on-web-publisher-controls/))
- **February 8, 2024:** After Bard is renamed, Google's documentation lists Gemini Apps as a product affected by Google-Extended. ([Source](https://developers.google.com/search/updates#updated-the-description-of-the-google-extended-product-token_1))
- **June 10, 2024:** Apple introduces the foundation models behind Apple Intelligence. Publishers can opt out of training use, according to Apple's documentation via Applebot-Extended. ([Source](https://machinelearning.apple.com/research/introducing-apple-foundation-models))
- **July 3, 2024:** Cloudflare lets customers block AI bots, scrapers and crawlers with a single click. ([Source](https://blog.cloudflare.com/declaring-your-aindependence-block-ai-bots-scrapers-and-crawlers-with-a-single-click/))
- **July 25, 2024:** With the SearchGPT prototype, OpenAI separates search from training. Sites can appear in search results even if they opt out of AI training. ([Source](https://openai.com/index/searchgpt-prototype/))
- **August 20, 2024:** Google adds Google-CloudVertexBot to its list of crawlers. It crawls sites at the owner's request to build Vertex AI Agents. ([Source](https://developers.google.com/search/updates#introducing-the-google-cloudvertexbot-crawler))
- **April 25, 2025:** Based on publisher feedback, Google clarifies the description of the Google-Extended token. ([Source](https://developers.google.com/search/updates#updated-the-description-of-the-google-extended-product-token))
- **May 15, 2025:** Cloudflare proposes that bots and agents cryptographically sign their requests, based on HTTP Message Signatures (RFC 9421). ([Source](https://blog.cloudflare.com/web-bot-auth/))
- **July 1, 2025:** Cloudflare changes its default to block AI crawlers unless site owners decide otherwise. ([Source](https://blog.cloudflare.com/content-independence-day-no-ai-crawl-without-compensation/))
- **July 1, 2025:** Cloudflare launches pay per crawl as a private beta. Crawlers either pay per request or receive a 402 status code with a price. ([Source](https://blog.cloudflare.com/introducing-pay-per-crawl/))

## Not to be confused with

- [OAI-SearchBot](https://seo-geo-glossary.com/en/glossary/oai-searchbot/index.md): surfaces websites in ChatGPT's search results. Sites that want to appear there need to allow it. According to OpenAI, the settings for the two bots are independent.

## Related terms

- [OAI-SearchBot](https://seo-geo-glossary.com/en/glossary/oai-searchbot/index.md)
- [ChatGPT-User](https://seo-geo-glossary.com/en/glossary/chatgpt-user/index.md)
- [AI crawler](https://seo-geo-glossary.com/en/glossary/ai-crawler/index.md)
- [Training crawler](https://seo-geo-glossary.com/en/glossary/training-crawler/index.md)
- [robots.txt](https://seo-geo-glossary.com/en/glossary/robots-txt/index.md)
- [User agent](https://seo-geo-glossary.com/en/glossary/user-agent/index.md)
- [Crawler IP ranges](https://seo-geo-glossary.com/en/glossary/crawler-ip-ranges/index.md)
- [Bot verification](https://seo-geo-glossary.com/en/glossary/bot-verification/index.md)

## More on the searchVIU blog

- [AI Crawlers & JavaScript Rendering](https://www.searchviu.com/en/ai-crawlers-javascript-rendering/)

## Sources

- [Overview of OpenAI crawlers – OpenAI](https://developers.openai.com/api/docs/bots)
- [GPTBot IP ranges (JSON) – OpenAI](https://openai.com/gptbot.json)
- [OpenAI's ChatGPT New Web Crawler – GPTBot – Search Engine Roundtable, 2023-08-07 (secondary source for the launch date)](https://www.seroundtable.com/openais-chatgpt-web-crawler-gptbot-35835.html)
- [GPTBot – how does it scrape JavaScript SPAs? – OpenAI Developer Community, September 2023](https://community.openai.com/t/gptbot-how-does-it-scrape-javascript-spas/402391)

---

Last updated: October 8, 2026
URL: https://seo-geo-glossary.com/en/glossary/gptbot/