GPTBot is OpenAI’s web crawler for collecting publicly available content that may be used to train its generative AI models. Site owners can allow or block it in robots.txt with the user-agent token GPTBot. According to reports, it became publicly known in August 2023.
At a glance
- Operator
- OpenAI
- Purpose
- Training generative AI foundation models
- robots.txt token
GPTBot- Respects robots.txt
- Disallowing it signals content should not be used for training
- IP list
openai.com/gptbot.json
Last checked:
What does GPTBot do?
OpenAI says GPTBot is used to make its generative AI foundation models more useful and safe. Disallowing GPTBot signals that a site’s content should not be used for training those models.
OpenAI runs other bots for other jobs. OAI-SearchBot surfaces websites in ChatGPT’s search features, and ChatGPT-User fetches pages when a user takes an action in ChatGPT or a custom GPT. OpenAI states that each setting is independent of the others. Blocking GPTBot therefore does not remove a site from ChatGPT search.
| Bot | Operator | Purpose | robots.txt token | robots.txt according to the operator | IP list published? |
|---|---|---|---|---|---|
| Google-Extended | Controls Gemini training and grounding (no crawler of its own) | Google-Extended | robots.txt token only; fetching is done by existing Google crawlers | n/a (no user agent of its own) | |
| GPTBot (this term) | OpenAI | Training | GPTBot | Disallowing it signals content should not be used to train foundation models | yes (openai.) |
| OAI-SearchBot | OpenAI | Search (ChatGPT) | OAI-SearchBot | opted-out sites are not shown in ChatGPT search answers | yes (openai.) |
| ChatGPT-User | OpenAI | User-triggered fetch | ChatGPT-User | robots.txt rules may not apply, says OpenAI (user-initiated) | yes (openai.) |
| ClaudeBot | Anthropic | Training | ClaudeBot | yes, Anthropic says its bots honor robots.txt | yes (claude.) |
| Claude-SearchBot | Anthropic | Search (Claude) | Claude-SearchBot | yes, Anthropic says its bots honor robots.txt | yes (claude.) |
| PerplexityBot | Perplexity | Search (Perplexity), not used for model training | PerplexityBot | controllable via robots.txt; the docs contain no explicit compliance statement | yes (perplexity.) |
| Applebot-Extended | Apple | Controls AI training (does not crawl, says Apple) | Applebot-Extended | robots.txt token only, applies to content Applebot already crawled | n/a (no crawler of its own) |
| CCBot | Common Crawl | Open web crawl archive | CCBot | can be blocked via robots.txt, says Common Crawl | yes (index.) |
| Meta-ExternalAgent | Meta | Training foundation AI models, improving products | meta-externalagent | can be blocked via robots.txt, says Meta | not stated by the operator |
A selection of ten AI bots from major operators, not a complete list. All values come from each operator's own documentation (as of 8 October 2026). The table does not show whether a bot actually complies with robots.txt; it only reflects what operators state. User agents can be spoofed; published IP lists or reverse DNS identify genuine requests. Sources: OpenAI: Overview of OpenAI Crawlers; Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler? (updated 2026-04-07); Perplexity: Perplexity Crawlers; Google: Google's common crawlers (last updated 2026-07-14); Google: Overview of Google crawlers and fetchers (last updated 2026-06-12); Google: Verifying Google crawlers and fetchers (IP range files); Apple: About Applebot (2026-09-04); Meta: Meta Web Crawlers; Common Crawl: CCBot. Last checked: .
How do you identify GPTBot?
As of October 2026, OpenAI documents this user-agent string:
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot
The version number changes over time, so filters should match the token GPTBot. Any client can fake a user agent. OpenAI therefore publishes GPTBot’s IP ranges at openai.com/gptbot.json. Comparing the IP address in the server logs with that list shows whether a request really came from OpenAI.
- The IETF publishes the Robots Exclusion Protocol as RFC 9309 on the Standards Track. It defines how crawlers interpret the rules in a robots.txt file.Source: rfc-editor.org
- OpenAI introduces web crawler permissions for AI. Site owners can use them to state whether their content may be used for AI training.Source: openai.com
- Google introduces Google-Extended. This robots.txt token controls whether content helps improve Bard and the Vertex AI generative APIs.Source: blog.google
- After Bard is renamed, Google's documentation lists Gemini Apps as a product affected by Google-Extended.Source: developers.google.com
- Apple introduces the foundation models behind Apple Intelligence. Publishers can opt out of training use, according to Apple's documentation via Applebot-Extended.Source: machinelearning.apple.com
- Cloudflare lets customers block AI bots, scrapers and crawlers with a single click.Source: blog.cloudflare.com
- With the SearchGPT prototype, OpenAI separates search from training. Sites can appear in search results even if they opt out of AI training.Source: openai.com
- Google adds Google-CloudVertexBot to its list of crawlers. It crawls sites at the owner's request to build Vertex AI Agents.Source: developers.google.com
- Based on publisher feedback, Google clarifies the description of the Google-Extended token.Source: developers.google.com
- Cloudflare proposes that bots and agents cryptographically sign their requests, based on HTTP Message Signatures (RFC 9421).Source: blog.cloudflare.com
- Cloudflare changes its default to block AI crawlers unless site owners decide otherwise.Source: blog.cloudflare.com
- Cloudflare launches pay per crawl as a private beta. Crawlers either pay per request or receive a 402 status code with a price.Source: blog.cloudflare.com
OpenAI only says “last summer” for its crawler permissions (post of May 7, 2024); the often-cited August 2023 date for GPTBot comes from press reports. ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, OAI-SearchBot, ChatGPT-User, Meta-ExternalAgent and CCBot are documented by their operators, but without a verifiable launch date. Last checked: .
Example
Block GPTBot but keep ChatGPT search access:
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
Block only certain directories for GPTBot:
User-agent: GPTBot
Disallow: /account/
Disallow: /internal/
Common mistakes
- Confusing GPTBot with the search bot. To appear in ChatGPT search, OAI-SearchBot needs access. The GPTBot rule only concerns training.
- Filtering by user agent alone. Spoofed requests then show up as GPTBot in reports or get blocked by mistake. The IP list separates real from fake requests.
- Overlooking blanket blocks at the host or CDN. Bot protection settings can block AI crawlers without anything appearing in robots.txt (assessment).
FAQ
What is a GPT bot? In the SEO context, GPTBot is OpenAI’s training crawler. The term is also used loosely for chatbots built on GPT models, which is a different thing.
Does blocking GPTBot remove my site from ChatGPT? Not according to OpenAI. ChatGPT search relies on OAI-SearchBot, and the settings for each bot are independent.
Does GPTBot render JavaScript? OpenAI does not say. Content that only appears after JavaScript runs may not be captured, so important text should be in the HTML the server delivers.
More on the searchVIU blog
Sources
- Overview of OpenAI crawlers – OpenAIdevelopers.openai.com
- GPTBot IP ranges (JSON) – OpenAIopenai.com
- OpenAI's ChatGPT New Web Crawler – GPTBot – Search Engine Roundtable, 2023-08-07 (secondary source for the launch date)seroundtable.com
- GPTBot – how does it scrape JavaScript SPAs? – OpenAI Developer Community, September 2023community.openai.com
Last updated: