Deutsch
Identifying & managing bots

Fake Googlebot

A fake Googlebot is a client that identifies itself as Googlebot in the user agent but does not come from Google. Operators use the disguise to get past access rules that let search engine crawlers through, or to make scraping and probing look less suspicious.

Why the user agent proves nothing

The user agent is a free-text HTTP header. Any client can write “Googlebot” into it. For real Google crawlers, the verification page points to the IP address and its reverse DNS name rather than the user agent string.

How impostors are detected

Google describes two ways. In the manual way, a reverse DNS lookup on the IP from your log must return a name ending in googlebot.com, google.com or googleusercontent.com, and a forward lookup of that name must return the same IP. In the automatic way, you match the IP against the crawler ranges Google publishes as JSON.

Rule sets on the web server side use the same idea. The Fake Bot plugin for the OWASP Core Rule Set compares the user agent of large search engines with a reverse DNS lookup. It adds to an anomaly score and does not block on its own.

Is it really Googlebot?Five-step flow: take the IP address from the log file, run a reverse DNS lookup, check that the hostname ends in googlebot.com, google.com or googleusercontent.com, run a forward DNS lookup on that hostname, check that the same IP comes back. Only if both checks pass does the request come from Google. Alternatively, the IP is matched against the JSON lists Google publishes.1 · IP address from the log file66.249.66.12 · Reverse DNS lookupcrawl-66-249-66-1.googlebot.com3 · Hostname ends ingooglebot.com, google.com, googleusercontent.com?yes4 · Forward DNS lookup on the hostname66.249.66.15 · Same IP as in the log file?yesRequest comes from Googlenonot Googlenonot GoogleOr automaticallyMatch the IP against Google’sJSON lists (CIDR format):common-crawlers.jsonspecial-crawlers.jsonuser-triggered-fetchers.jsonuser-triggered-fetchers-google.jsonuser-triggered-agents.jsonOther crawlers often spoofthe Googlebot user agent.It proves nothing on its own.
Both DNS lookups have to match. For large-scale lookups, Google recommends matching against its published IP lists automatically. Sample values from Google’s documentation. Sources: Google Crawling: Verify requests from Google crawlers and fetchers; Google Search Central: What is Googlebot.

Example

Two lines from a log (sample data) carry the same user agent but come from different addresses:

Text
192.0.2.31 - - [06/Oct/2026:08:01:10 +0000] "GET /admin/ HTTP/1.1" 403 199 "-" "Googlebot/2.1"
198.51.100.7 - - [06/Oct/2026:08:01:12 +0000] "GET /products/ HTTP/1.1" 200 20412 "-" "Googlebot/2.1"

Only checking the IP addresses shows which of the two belongs to Google.

Common mistakes

  • Tying exceptions to the user agent alone. Rules that exempt “Googlebot” from rate limits, logins or bot protection then apply to anyone who writes that text into the header (a conclusion from the freely settable header).
  • Skipping the forward lookup. Google requires that the forward lookup of the host name returns the original IP, not just a matching host name in the reverse lookup.
  • Counting fake requests in the analysis. If a log analysis counts every line with a Googlebot user agent, impostors end up in the crawl statistics (practical experience).

FAQ

Can I block everything that calls itself Googlebot? A block based on the user agent alone also hits real requests if the imitation looks identical. A decision based on verification is safer.

How do I verify Googlebot? Use reverse and forward DNS or Google’s published IP ranges.

Sources

  1. Verifying Google crawlers and fetchers – Google Search Centraldevelopers.google.com
  2. Introducing the Fake Bot Plugin – OWASP CRScoreruleset.org
  3. What is Googlebot (impostor)? – Crawler Zoo (secondary)crawlerzoo.com

Last updated: