Deutsch
Identifying & managing bots

Bot verification

Bot verification is the check whether a crawler really belongs to the operator named in its user agent. In SEO and server work it usually means verifying Googlebot and other crawlers in your own logs, not the bot checks of chat platforms or CAPTCHAs.

Why verify?

The user agent is a text value that any client can set. It is not proof of identity. If you grant crawlers special access, exempt them from rate limits or analyze log files, you want to separate real crawler requests from imitations (see fake Googlebot).

Methods

Reverse and forward DNS. Google describes a two-step check. Run a reverse DNS lookup on the IP address from your log. The host name must end in googlebot.com, google.com or googleusercontent.com. Then run a forward lookup on that host name and confirm that it resolves to the original IP address. Both steps belong together.

Google documents which host names to expect per crawler group:

Crawler group Reverse DNS pattern
Common crawlers (e.g. Googlebot) crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com
Special-case crawlers rate-limited-proxy-***-***-***-***.google.com
User-triggered fetchers ***-***-***-***.gae.googleusercontent.com or google-proxy-***-***-***-***.google.com

Published IP ranges. Google publishes crawler IP ranges as JSON files in CIDR format, split by crawler group, for example common-crawlers.json for Googlebot. Since March 2026 the files live under /static/crawling/ipranges/ on developers.google.com. Matching an IP address against these lists can be automated (see crawler IP ranges).

Signatures. Cloudflare verifies bots through a cryptographic signature (Web Bot Auth, for which the IETF has set up a working group called webbotauth) or through a published IP list combined with a stable user agent or reverse DNS. Cloudflare also requires that the bot obeys robots.txt and keeps request rates reasonable.

Is it really Googlebot?Five-step flow: take the IP address from the log file, run a reverse DNS lookup, check that the hostname ends in googlebot.com, google.com or googleusercontent.com, run a forward DNS lookup on that hostname, check that the same IP comes back. Only if both checks pass does the request come from Google. Alternatively, the IP is matched against the JSON lists Google publishes.1 · IP address from the log file66.249.66.12 · Reverse DNS lookupcrawl-66-249-66-1.googlebot.com3 · Hostname ends ingooglebot.com, google.com, googleusercontent.com?yes4 · Forward DNS lookup on the hostname66.249.66.15 · Same IP as in the log file?yesRequest comes from Googlenonot Googlenonot GoogleOr automaticallyMatch the IP against Google’sJSON lists (CIDR format):common-crawlers.jsonspecial-crawlers.jsonuser-triggered-fetchers.jsonuser-triggered-fetchers-google.jsonuser-triggered-agents.jsonOther crawlers often spoofthe Googlebot user agent.It proves nothing on its own.
Both DNS lookups have to match. For large-scale lookups, Google recommends matching against its published IP lists automatically. Sample values from Google’s documentation. Sources: Google Crawling: Verify requests from Google crawlers and fetchers; Google Search Central: What is Googlebot.

Example

Checking a suspicious IP address (placeholder values):

Shell
host 192.0.2.10
host crawl-example.googlebot.com

Only if the first command returns a host name with a matching ending and the second resolves back to 192.0.2.10 does the request come from Google.

Common mistakes

  • Trusting the user agent string alone.
  • Running only the reverse lookup and skipping the forward check.
  • Expecting every Google request to follow robots.txt. Google states that user-triggered fetchers generally ignore robots.txt rules because a user requested the fetch.
  • Scripts that still point to the old IP file location under /search/apis/ipranges/.

FAQ

How do I verify Googlebot? Use the reverse and forward DNS check above, or compare the IP address with Google’s published ranges.

Sources

  1. Verifying Google crawlers and fetchers – Google Crawling Infrastructuredevelopers.google.com
  2. New location for the Google crawlers' IP range files – Google Search Central Blog (Mar 31, 2026)developers.google.com
  3. Google user-triggered fetchers – Google Search Centraldevelopers.google.com
  4. Verified bots – Cloudflare Docsdevelopers.cloudflare.com
  5. Web Bot Auth Working Group – IETF Datatrackerdatatracker.ietf.org

Last updated: