# Google-CloudVertexBot

Category: [AI Bots & Crawlers](https://seo-geo-glossary.com/en/glossary/category/ai-bots-crawlers/) › [Other AI bots](https://seo-geo-glossary.com/en/glossary/category/ai-bots-crawlers/other-ai-bots/)

**Google-CloudVertexBot** is a Google crawler that fetches a website when the site owner has requested it, to build Vertex AI agents. It is one of Google's common crawlers and has no effect on Google Search.

## What is Google-CloudVertexBot used for?

Google's crawler documentation says crawling preferences addressed to the `Google-CloudVertexBot` user agent affect crawls requested by the site owners for building Vertex AI agents. Vertex AI Agent Builder is Google Cloud's product for such agents. Search Engine Journal reported in August 2024 that the bot only crawls on the site owner's request, so a site that never asked for it should not see it.

It is separate from Googlebot, which crawls for Google Search, and from Google-Extended, which is a robots.txt token for AI training and grounding with no user agent of its own.

## Which robots.txt tokens apply?

Google lists two tokens for this crawler, `Google-CloudVertexBot` and `Googlebot`.

## Example

```robots
User-agent: Google-CloudVertexBot
Disallow: /internal/
```

The rule applies to this crawler. Rules for other Google crawlers stay as they are.

## At a glance

- **Operator:** Google
- **Purpose:** Crawls requested by the site owner for Vertex AI Agents
- **robots.txt token:** Google-CloudVertexBot, Googlebot
- **Respects robots.txt:** yes, Google's common crawlers always respect robots.txt for automatic crawls
- **IP list:** yes (common-crawlers.json)
- **Docs:** [Google: Common Crawlers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers)

Last checked: October 8, 2026

## AI bots compared

| Bot | Operator | Purpose | robots.txt token | robots.txt according to the operator | IP list published? |
|---|---|---|---|---|---|
| Google-Extended | Google | Controls Gemini training and grounding (no crawler of its own) | Google-Extended | robots.txt token only; fetching is done by existing Google crawlers | n/a (no user agent of its own) |
| GPTBot | OpenAI | Training | GPTBot | Disallowing it signals content should not be used to train foundation models | yes (openai.com/gptbot.json) |
| OAI-SearchBot | OpenAI | Search (ChatGPT) | OAI-SearchBot | opted-out sites are not shown in ChatGPT search answers | yes (openai.com/searchbot.json) |
| ChatGPT-User | OpenAI | User-triggered fetch | ChatGPT-User | robots.txt rules may not apply, says OpenAI (user-initiated) | yes (openai.com/chatgpt-user.json) |
| ClaudeBot | Anthropic | Training | ClaudeBot | yes, Anthropic says its bots honor robots.txt | yes (claude.com/crawling/bots.json) |
| Claude-SearchBot | Anthropic | Search (Claude) | Claude-SearchBot | yes, Anthropic says its bots honor robots.txt | yes (claude.com/crawling/bots.json) |
| PerplexityBot | Perplexity | Search (Perplexity), not used for model training | PerplexityBot | controllable via robots.txt; the docs contain no explicit compliance statement | yes (perplexity.com/perplexitybot.json) |
| Applebot-Extended | Apple | Controls AI training (does not crawl, says Apple) | Applebot-Extended | robots.txt token only, applies to content Applebot already crawled | n/a (no crawler of its own) |
| CCBot | Common Crawl | Open web crawl archive | CCBot | can be blocked via robots.txt, says Common Crawl | yes (index.commoncrawl.org/ccbot.json) |
| Meta-ExternalAgent | Meta | Training foundation AI models, improving products | meta-externalagent | can be blocked via robots.txt, says Meta | not stated by the operator |

Sources: [OpenAI: Overview of OpenAI Crawlers](https://developers.openai.com/api/docs/bots); [Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler? (updated 2026-04-07)](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler); [Perplexity: Perplexity Crawlers](https://docs.perplexity.ai/guides/bots); [Google: Google's common crawlers (last updated 2026-07-14)](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers); [Google: Overview of Google crawlers and fetchers (last updated 2026-06-12)](https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers); [Google: Verifying Google crawlers and fetchers (IP range files)](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests); [Apple: About Applebot (2026-09-04)](https://support.apple.com/en-us/119829); [Meta: Meta Web Crawlers](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/); [Common Crawl: CCBot](https://commoncrawl.org/ccbot). Last checked: October 8, 2026

## Related terms

- [Googlebot](https://seo-geo-glossary.com/en/glossary/googlebot/index.md)
- [Google-Extended](https://seo-geo-glossary.com/en/glossary/google-extended/index.md)
- [AI crawler](https://seo-geo-glossary.com/en/glossary/ai-crawler/index.md)
- [robots.txt](https://seo-geo-glossary.com/en/glossary/robots-txt/index.md)

## Sources

- [Google Common Crawlers – Google Search Central](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers)
- [Google's new AI crawler for Vertex AI – Search Engine Journal (21.08.2024)](https://www.searchenginejournal.com/google-vertex-ai-crawler/524956/)

---

Last updated: October 8, 2026
URL: https://seo-geo-glossary.com/en/glossary/google-cloudvertexbot/