# Google-Extended

Category: [AI Search & GEO](https://seo-geo-glossary.com/en/glossary/category/ai-search-geo/) › [Controls & opt-out](https://seo-geo-glossary.com/en/glossary/category/ai-search-geo/controls-opt-out/)

**Google-Extended** is a robots.txt product token that lets site owners control whether content Google has crawled may be used for training Gemini models and for grounding in Gemini apps and Vertex AI. Google announced it on September 28, 2023.

## How does Google-Extended work?

Google-Extended is a standalone token. According to Google's crawler documentation, it has no separate HTTP user agent string, and crawling runs through Google's existing user agents. For that reason it never appears as its own entry in server logs.

The documentation also states that Google-Extended does not affect a site's inclusion in Google Search and is not used as a ranking signal. Google describes AI Overviews and AI Mode as features of Search. For these, it lists no requirements beyond being indexed and eligible for snippets. So Google-Extended most likely does not govern them (assessment, not a statement from Google).

## Example

A robots.txt entry following Google's documentation:

```robots
user-agent: Google-Extended
allow: /archive/1Q84
disallow: /archive/
```

Everything under `/archive/` is then off limits for the Gemini purposes above, except the one allowed page.

## Common questions

**Does blocking Google-Extended hurt rankings?**
Not according to Google. The token is not a ranking signal and does not change inclusion in Search.

**Does it block other AI crawlers?**
No. It applies to Google only. Other providers publish their own bots and tokens, for example OpenAI's GPTBot.

## At a glance

- **Operator:** Google
- **Purpose:** Controls use of crawled content for Gemini training and grounding
- **robots.txt token:** Google-Extended
- **Respects robots.txt:** robots.txt token only, fetching uses Google's existing crawlers
- **Since:** 2023-09-28
- **Docs:** [Google's common crawlers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers)

Last checked: October 8, 2026

## AI bots compared

| Bot | Operator | Purpose | robots.txt token | robots.txt according to the operator | IP list published? |
|---|---|---|---|---|---|
| Google-Extended | Google | Controls Gemini training and grounding (no crawler of its own) | Google-Extended | robots.txt token only; fetching is done by existing Google crawlers | n/a (no user agent of its own) |
| GPTBot | OpenAI | Training | GPTBot | Disallowing it signals content should not be used to train foundation models | yes (openai.com/gptbot.json) |
| OAI-SearchBot | OpenAI | Search (ChatGPT) | OAI-SearchBot | opted-out sites are not shown in ChatGPT search answers | yes (openai.com/searchbot.json) |
| ChatGPT-User | OpenAI | User-triggered fetch | ChatGPT-User | robots.txt rules may not apply, says OpenAI (user-initiated) | yes (openai.com/chatgpt-user.json) |
| ClaudeBot | Anthropic | Training | ClaudeBot | yes, Anthropic says its bots honor robots.txt | yes (claude.com/crawling/bots.json) |
| Claude-SearchBot | Anthropic | Search (Claude) | Claude-SearchBot | yes, Anthropic says its bots honor robots.txt | yes (claude.com/crawling/bots.json) |
| PerplexityBot | Perplexity | Search (Perplexity), not used for model training | PerplexityBot | controllable via robots.txt; the docs contain no explicit compliance statement | yes (perplexity.com/perplexitybot.json) |
| Applebot-Extended | Apple | Controls AI training (does not crawl, says Apple) | Applebot-Extended | robots.txt token only, applies to content Applebot already crawled | n/a (no crawler of its own) |
| CCBot | Common Crawl | Open web crawl archive | CCBot | can be blocked via robots.txt, says Common Crawl | yes (index.commoncrawl.org/ccbot.json) |
| Meta-ExternalAgent | Meta | Training foundation AI models, improving products | meta-externalagent | can be blocked via robots.txt, says Meta | not stated by the operator |

Sources: [OpenAI: Overview of OpenAI Crawlers](https://developers.openai.com/api/docs/bots); [Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler? (updated 2026-04-07)](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler); [Perplexity: Perplexity Crawlers](https://docs.perplexity.ai/guides/bots); [Google: Google's common crawlers (last updated 2026-07-14)](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers); [Google: Overview of Google crawlers and fetchers (last updated 2026-06-12)](https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers); [Google: Verifying Google crawlers and fetchers (IP range files)](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests); [Apple: About Applebot (2026-09-04)](https://support.apple.com/en-us/119829); [Meta: Meta Web Crawlers](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/); [Common Crawl: CCBot](https://commoncrawl.org/ccbot). Last checked: October 8, 2026

## Google crawlers compared

| Crawler | robots.txt token | Purpose | Effect of blocking on Google Search | Shows up in logs as |
|---|---|---|---|---|
| Googlebot Smartphone | Googlebot | Crawling for Google Search (incl. Discover), Images, Video, News; main crawler for most sites | pages are no longer crawled for Search | Googlebot/2.1 with an Android identifier |
| Googlebot Desktop | Googlebot | same as Smartphone, simulates a desktop user | same as Smartphone (shared token, cannot be blocked separately) | Googlebot/2.1 without a mobile identifier |
| Storebot-Google | Storebot-Google | Shopping: product discovery and indexing | affects Shopping surfaces incl. the Shopping tab in Search | Storebot-Google/1.0 |
| GoogleOther | GoogleOther | generic crawler for product teams, e.g. one-off research and development crawls | doesn't affect any specific product, says Google | GoogleOther |
| Google-InspectionTool | Google-InspectionTool, Googlebot | Testing tools: Rich Results Test, URL Inspection in Search Console | none (affects only the testing tools) | Google-InspectionTool/1.0 |
| Google-CloudVertexBot | Google-CloudVertexBot, Googlebot | Crawls requested by the site owner for Vertex AI Agents | none | Google-CloudVertexBot |
| Google-Extended | Google-Extended | Controls whether content is used for Gemini training and grounding | none; Google says it doesn't affect inclusion in Search and isn't a ranking signal | not at all (no user agent string of its own) |

Sources: [Google: Google's common crawlers (last updated 2026-07-14)](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers); [Google: Overview of Google crawlers and fetchers (last updated 2026-06-12)](https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers); [Google Search Central: Googlebot (last updated 2026-02-03)](https://developers.google.com/search/docs/crawling-indexing/googlebot). Last checked: October 8, 2026

## AI bots: crawlers, tokens and controls

- **September 2022:** The IETF publishes the Robots Exclusion Protocol as RFC 9309 on the Standards Track. It defines how crawlers interpret the rules in a robots.txt file. ([Source](https://www.rfc-editor.org/rfc/rfc9309.html))
- **2023:** OpenAI introduces web crawler permissions for AI. Site owners can use them to state whether their content may be used for AI training. ([Source](https://openai.com/index/approach-to-data-and-ai/))
- **September 28, 2023:** Google introduces Google-Extended. This robots.txt token controls whether content helps improve Bard and the Vertex AI generative APIs. ([Source](https://blog.google/technology/ai/an-update-on-web-publisher-controls/))
- **February 8, 2024:** After Bard is renamed, Google's documentation lists Gemini Apps as a product affected by Google-Extended. ([Source](https://developers.google.com/search/updates#updated-the-description-of-the-google-extended-product-token_1))
- **June 10, 2024:** Apple introduces the foundation models behind Apple Intelligence. Publishers can opt out of training use, according to Apple's documentation via Applebot-Extended. ([Source](https://machinelearning.apple.com/research/introducing-apple-foundation-models))
- **July 3, 2024:** Cloudflare lets customers block AI bots, scrapers and crawlers with a single click. ([Source](https://blog.cloudflare.com/declaring-your-aindependence-block-ai-bots-scrapers-and-crawlers-with-a-single-click/))
- **July 25, 2024:** With the SearchGPT prototype, OpenAI separates search from training. Sites can appear in search results even if they opt out of AI training. ([Source](https://openai.com/index/searchgpt-prototype/))
- **August 20, 2024:** Google adds Google-CloudVertexBot to its list of crawlers. It crawls sites at the owner's request to build Vertex AI Agents. ([Source](https://developers.google.com/search/updates#introducing-the-google-cloudvertexbot-crawler))
- **April 25, 2025:** Based on publisher feedback, Google clarifies the description of the Google-Extended token. ([Source](https://developers.google.com/search/updates#updated-the-description-of-the-google-extended-product-token))
- **May 15, 2025:** Cloudflare proposes that bots and agents cryptographically sign their requests, based on HTTP Message Signatures (RFC 9421). ([Source](https://blog.cloudflare.com/web-bot-auth/))
- **July 1, 2025:** Cloudflare changes its default to block AI crawlers unless site owners decide otherwise. ([Source](https://blog.cloudflare.com/content-independence-day-no-ai-crawl-without-compensation/))
- **July 1, 2025:** Cloudflare launches pay per crawl as a private beta. Crawlers either pay per request or receive a 402 status code with a price. ([Source](https://blog.cloudflare.com/introducing-pay-per-crawl/))

## Not to be confused with

- [Googlebot](https://seo-geo-glossary.com/en/glossary/googlebot/index.md): fetches pages for Google Search and appears in server logs with its own user agent. The token has no user agent of its own and does not affect inclusion in Search.

## Related terms

- [Googlebot](https://seo-geo-glossary.com/en/glossary/googlebot/index.md)
- [robots.txt](https://seo-geo-glossary.com/en/glossary/robots-txt/index.md)
- [Training crawler](https://seo-geo-glossary.com/en/glossary/training-crawler/index.md)
- [Blocking AI bots](https://seo-geo-glossary.com/en/glossary/blocking-ai-bots/index.md)
- [Training data](https://seo-geo-glossary.com/en/glossary/training-data/index.md)
- [Applebot / Applebot-Extended](https://seo-geo-glossary.com/en/glossary/applebot-applebot-extended/index.md)
- [AI Overviews](https://seo-geo-glossary.com/en/glossary/ai-overviews/index.md)

## Sources

- [Google's common crawlers (Google-Extended) – Google Crawling Infrastructure](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers)
- [An update on web publisher controls – Google Blog (Sep 28, 2023)](https://blog.google/innovation-and-ai/products/an-update-on-web-publisher-controls/)
- [AI features and your website – Google Search Central](https://developers.google.com/search/docs/appearance/ai-features)

---

Last updated: October 8, 2026
URL: https://seo-geo-glossary.com/en/glossary/google-extended/