# Googlebot

Category: [Crawling & Indexing](https://seo-geo-glossary.com/en/glossary/category/crawling-indexing/) › [Crawlers & bots](https://seo-geo-glossary.com/en/glossary/category/crawling-indexing/crawlers-bots/) · Also known as: Google crawler

**Googlebot** is the web crawler of Google Search. It fetches pages and resources such as CSS and JavaScript files so that Google can process, render, and index them for search. The name covers two crawlers, Googlebot Smartphone and Googlebot Desktop.

## How does Googlebot work?

Googlebot finds URLs through links and sitemaps, checks robots.txt, and then downloads what it is allowed to fetch. Both variants obey the same product token, `Googlebot`, in robots.txt. A rule for `Googlebot` always applies to both, and there is no way to target only one of them. Because Google mainly indexes the mobile version of content, most requests come from Googlebot Smartphone.

On server load, Google says Googlebot should not access most sites more than once every few seconds on average, though short bursts can look faster. If a server cannot keep up, site owners can reduce the crawl rate.

For Google Search, Googlebot fetches the first 2 MB of a supported file and the first 64 MB of a PDF. According to Google's blog, the 2 MB include the HTTP header. Anything past the cutoff is ignored for indexing. Each resource referenced in the HTML, such as a CSS or JavaScript file, is fetched separately under the same limit. Other Google crawlers have their own limits, with 15 MB as the general default for Google's crawlers and fetchers. Some third-party sources still give 15 MB for Googlebot. Googlebot is one client of Google's shared crawling infrastructure, which also serves products like Google Shopping and AdSense under different crawler names.

## Example

The user-agent string of Googlebot Smartphone (the Chrome version changes over time):

```txt
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
```

A robots.txt rule for Googlebot:

```robots
User-agent: Googlebot
Disallow: /internal/
```

## How to verify a real Googlebot

Any client can send a Googlebot user-agent string, and scrapers often do. Google describes two checks. For single requests, run a reverse DNS lookup on the IP address: the hostname must end in `googlebot.com`, `google.com`, or `googleusercontent.com`. Then run a forward DNS lookup on that hostname and confirm it returns the same IP. For large volumes, match IP addresses automatically against the IP range files Google publishes as JSON. In log file analysis, filter out fake Googlebot requests before drawing conclusions.

## Common mistakes

- **Less content on the mobile version.** Google mainly indexes the mobile version. If the mobile page lacks content the desktop page has, Google's documentation says it can get less information from the page.
- **Different robots meta tags on mobile.** Google recommends the same robots meta tags on the mobile and desktop versions. A `noindex` on the mobile page alone affects indexing under mobile-first indexing.
- **Counting fake Googlebots.** Without a DNS or IP check, scrapers with a Googlebot user agent end up in the analysis. Google recommends reverse and forward DNS lookups or a match against its published IP ranges.

## FAQ

**Can you block Googlebot?**
Yes, with a `Disallow` rule for `Googlebot` in robots.txt. This stops crawling, but a blocked URL can still appear in the index if other pages link to it. To keep a page out of the index, use `noindex` and let Googlebot crawl the page so it sees the directive.

**How often does Googlebot crawl a site?**
Google does not publish a fixed schedule. Frequency depends on the site's server capacity and on how much Google wants to crawl it, for example because of new or frequently updated content.

**Is Googlebot the same as Google-Extended?**
No. Google-Extended is a robots.txt token that controls whether content Google crawls may be used to train Gemini models and for grounding in Gemini apps. It has no separate user agent; crawling happens with existing Google crawlers. Google states that it does not affect inclusion or ranking in Google Search.

## At a glance

- **Operator:** Google
- **Purpose:** crawling for Google Search (smartphone and desktop variants)
- **robots.txt token:** Googlebot (applies to both variants)
- **Respects robots.txt:** yes
- **IP list:** [common-crawlers.json](https://developers.google.com/static/crawling/ipranges/common-crawlers.json)
- **Docs:** [Google: Googlebot](https://developers.google.com/search/docs/crawling-indexing/googlebot)

Last checked: October 8, 2026

## Google crawlers compared

| Crawler | robots.txt token | Purpose | Effect of blocking on Google Search | Shows up in logs as |
|---|---|---|---|---|
| Googlebot Smartphone | Googlebot | Crawling for Google Search (incl. Discover), Images, Video, News; main crawler for most sites | pages are no longer crawled for Search | Googlebot/2.1 with an Android identifier |
| Googlebot Desktop | Googlebot | same as Smartphone, simulates a desktop user | same as Smartphone (shared token, cannot be blocked separately) | Googlebot/2.1 without a mobile identifier |
| Storebot-Google | Storebot-Google | Shopping: product discovery and indexing | affects Shopping surfaces incl. the Shopping tab in Search | Storebot-Google/1.0 |
| GoogleOther | GoogleOther | generic crawler for product teams, e.g. one-off research and development crawls | doesn't affect any specific product, says Google | GoogleOther |
| Google-InspectionTool | Google-InspectionTool, Googlebot | Testing tools: Rich Results Test, URL Inspection in Search Console | none (affects only the testing tools) | Google-InspectionTool/1.0 |
| Google-CloudVertexBot | Google-CloudVertexBot, Googlebot | Crawls requested by the site owner for Vertex AI Agents | none | Google-CloudVertexBot |
| Google-Extended | Google-Extended | Controls whether content is used for Gemini training and grounding | none; Google says it doesn't affect inclusion in Search and isn't a ranking signal | not at all (no user agent string of its own) |

Sources: [Google: Google's common crawlers (last updated 2026-07-14)](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers); [Google: Overview of Google crawlers and fetchers (last updated 2026-06-12)](https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers); [Google Search Central: Googlebot (last updated 2026-02-03)](https://developers.google.com/search/docs/crawling-indexing/googlebot). Last checked: October 8, 2026

*Diagram: Is it really Googlebot?.* Both DNS lookups have to match. For large-scale lookups, Google recommends matching against its published IP lists automatically. Sample values from Google’s documentation.

## Not to be confused with

- [Google-Extended](https://seo-geo-glossary.com/en/glossary/google-extended/index.md): is a robots.txt token without a crawler of its own. It controls whether content Google crawls may be used for Gemini models and has no effect on Google Search.

## Related terms

- [Googlebot Smartphone](https://seo-geo-glossary.com/en/glossary/googlebot-smartphone/index.md)
- [Fake Googlebot](https://seo-geo-glossary.com/en/glossary/fake-googlebot/index.md)

## Sources

- [Googlebot – Google Search Central (last updated 2026-02-03)](https://developers.google.com/search/docs/crawling-indexing/googlebot)
- [Inside Googlebot: demystifying crawling, fetching, and the bytes we process – Google Search Central Blog (March 2026)](https://developers.google.com/search/blog/2026/03/crawler-blog-post)
- [Overview of Google crawlers and fetchers – Google Crawling Infrastructure (last updated 2026-06-12)](https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers)
- [Google's common crawlers – Google Crawling Infrastructure](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers)
- [Verifying requests from Google crawlers and fetchers – Google Crawling Infrastructure (last updated 2026-03-20)](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests)
- [Crawl budget management – Google Crawling Infrastructure (last updated 2026-07-22)](https://developers.google.com/crawling/docs/crawl-budget)
- [Mobile-first indexing best practices – Google Search Central](https://developers.google.com/search/docs/crawling-indexing/mobile/mobile-sites-mobile-first-indexing)

---

Last updated: October 8, 2026
URL: https://seo-geo-glossary.com/en/glossary/googlebot/