Deutsch
Crawlers & bots

Googlebot

Also known as: Google crawler

Googlebot is the web crawler of Google Search. It fetches pages and resources such as CSS and JavaScript files so that Google can process, render, and index them for search. The name covers two crawlers, Googlebot Smartphone and Googlebot Desktop.

At a glance

Operator
Google
Purpose
crawling for Google Search (smartphone and desktop variants)
robots.txt token
Googlebot (applies to both variants)
Respects robots.txt
yes
IP list
common-crawlers.json
Docs
Google: Googlebot

Last checked:

How does Googlebot work?

Googlebot finds URLs through links and sitemaps, checks robots.txt, and then downloads what it is allowed to fetch. Both variants obey the same product token, Googlebot, in robots.txt. A rule for Googlebot always applies to both, and there is no way to target only one of them. Because Google mainly indexes the mobile version of content, most requests come from Googlebot Smartphone.

On server load, Google says Googlebot should not access most sites more than once every few seconds on average, though short bursts can look faster. If a server cannot keep up, site owners can reduce the crawl rate.

For Google Search, Googlebot fetches the first 2 MB of a supported file and the first 64 MB of a PDF. According to Google’s blog, the 2 MB include the HTTP header. Anything past the cutoff is ignored for indexing. Each resource referenced in the HTML, such as a CSS or JavaScript file, is fetched separately under the same limit. Other Google crawlers have their own limits, with 15 MB as the general default for Google’s crawlers and fetchers. Some third-party sources still give 15 MB for Googlebot. Googlebot is one client of Google’s shared crawling infrastructure, which also serves products like Google Shopping and AdSense under different crawler names.

Google crawlers compared
Crawlerrobots.txt tokenPurposeEffect of blocking on Google SearchShows up in logs as
Googlebot SmartphoneGooglebotCrawling for Google Search (incl. Discover), Images, Video, News; main crawler for most sitespages are no longer crawled for SearchGooglebot/2.1 with an Android identifier
Googlebot Desktop (this term)Googlebotsame as Smartphone, simulates a desktop usersame as Smartphone (shared token, cannot be blocked separately)Googlebot/2.1 without a mobile identifier
Storebot-GoogleStorebot-GoogleShopping: product discovery and indexingaffects Shopping surfaces incl. the Shopping tab in SearchStorebot-Google/1.0
GoogleOtherGoogleOthergeneric crawler for product teams, e.g. one-off research and development crawlsdoesn't affect any specific product, says GoogleGoogleOther
Google-InspectionToolGoogle-InspectionTool, GooglebotTesting tools: Rich Results Test, URL Inspection in Search Consolenone (affects only the testing tools)Google-InspectionTool/1.0
Google-CloudVertexBotGoogle-CloudVertexBot, GooglebotCrawls requested by the site owner for Vertex AI AgentsnoneGoogle-CloudVertexBot
Google-ExtendedGoogle-ExtendedControls whether content is used for Gemini training and groundingnone; Google says it doesn't affect inclusion in Search and isn't a ranking signalnot at all (no user agent string of its own)

Googlebot Smartphone and Desktop share the Googlebot token and cannot be targeted separately in robots.txt. Google says most Googlebot requests come from the mobile crawler. Google-Extended is a control token only; crawling uses existing Google user agents. All crawlers listed here are Google common crawlers, which always respect robots.txt for automatic crawls. Sources: Google: Google's common crawlers (last updated 2026-07-14); Google: Overview of Google crawlers and fetchers (last updated 2026-06-12); Google Search Central: Googlebot (last updated 2026-02-03). Last checked: .

Example

The user-agent string of Googlebot Smartphone (the Chrome version changes over time):

Text
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

A robots.txt rule for Googlebot:

robots.txt
User-agent: Googlebot
Disallow: /internal/

How to verify a real Googlebot

Any client can send a Googlebot user-agent string, and scrapers often do. Google describes two checks. For single requests, run a reverse DNS lookup on the IP address: the hostname must end in googlebot.com, google.com, or googleusercontent.com. Then run a forward DNS lookup on that hostname and confirm it returns the same IP. For large volumes, match IP addresses automatically against the IP range files Google publishes as JSON. In log file analysis, filter out fake Googlebot requests before drawing conclusions.

Is it really Googlebot?Five-step flow: take the IP address from the log file, run a reverse DNS lookup, check that the hostname ends in googlebot.com, google.com or googleusercontent.com, run a forward DNS lookup on that hostname, check that the same IP comes back. Only if both checks pass does the request come from Google. Alternatively, the IP is matched against the JSON lists Google publishes.1 · IP address from the log file66.249.66.12 · Reverse DNS lookupcrawl-66-249-66-1.googlebot.com3 · Hostname ends ingooglebot.com, google.com, googleusercontent.com?yes4 · Forward DNS lookup on the hostname66.249.66.15 · Same IP as in the log file?yesRequest comes from Googlenonot Googlenonot GoogleOr automaticallyMatch the IP against Google’sJSON lists (CIDR format):common-crawlers.jsonspecial-crawlers.jsonuser-triggered-fetchers.jsonuser-triggered-fetchers-google.jsonuser-triggered-agents.jsonOther crawlers often spoofthe Googlebot user agent.It proves nothing on its own.
Both DNS lookups have to match. For large-scale lookups, Google recommends matching against its published IP lists automatically. Sample values from Google’s documentation. Sources: Google Crawling: Verify requests from Google crawlers and fetchers; Google Search Central: What is Googlebot.

Common mistakes

  • Less content on the mobile version. Google mainly indexes the mobile version. If the mobile page lacks content the desktop page has, Google’s documentation says it can get less information from the page.
  • Different robots meta tags on mobile. Google recommends the same robots meta tags on the mobile and desktop versions. A noindex on the mobile page alone affects indexing under mobile-first indexing.
  • Counting fake Googlebots. Without a DNS or IP check, scrapers with a Googlebot user agent end up in the analysis. Google recommends reverse and forward DNS lookups or a match against its published IP ranges.

FAQ

Can you block Googlebot? Yes, with a Disallow rule for Googlebot in robots.txt. This stops crawling, but a blocked URL can still appear in the index if other pages link to it. To keep a page out of the index, use noindex and let Googlebot crawl the page so it sees the directive.

How often does Googlebot crawl a site? Google does not publish a fixed schedule. Frequency depends on the site’s server capacity and on how much Google wants to crawl it, for example because of new or frequently updated content.

Is Googlebot the same as Google-Extended? No. Google-Extended is a robots.txt token that controls whether content Google crawls may be used to train Gemini models and for grounding in Gemini apps. It has no separate user agent; crawling happens with existing Google crawlers. Google states that it does not affect inclusion or ranking in Google Search.

Sources

  1. Googlebot – Google Search Central (last updated 2026-02-03)developers.google.com
  2. Inside Googlebot: demystifying crawling, fetching, and the bytes we process – Google Search Central Blog (March 2026)developers.google.com
  3. Overview of Google crawlers and fetchers – Google Crawling Infrastructure (last updated 2026-06-12)developers.google.com
  4. Google's common crawlers – Google Crawling Infrastructuredevelopers.google.com
  5. Verifying requests from Google crawlers and fetchers – Google Crawling Infrastructure (last updated 2026-03-20)developers.google.com
  6. Crawl budget management – Google Crawling Infrastructure (last updated 2026-07-22)developers.google.com
  7. Mobile-first indexing best practices – Google Search Centraldevelopers.google.com

Last updated: