Deutsch
Servers, HTTP & caching

Reverse DNS lookup

Also known as: reverse DNS, rDNS, PTR lookup

A reverse DNS lookup (rDNS) is a DNS query that finds the hostname belonging to an IP address. A normal lookup goes from name to address. The reverse lookup goes from address to name, using a PTR (pointer) record. For SEO, it matters mainly as a way to check whether a request that claims to come from a search engine bot really does.

How does a reverse DNS lookup work?

Reverse records live in special zones. For IPv4, the address is written in reverse octet order under in-addr.arpa: the address 192.0.2.10 is queried as 10.2.0.192.in-addr.arpa. IPv6 uses the zone ip6.arpa. The owner of the IP block, usually a hosting provider or the crawler operator, decides whether to maintain PTR records, which are optional.

Verifying Googlebot

Google documents a check for requests in server logs:

  1. Run a reverse DNS lookup on the IP address from the log.
  2. Check that the hostname belongs to googlebot.com, google.com or googleusercontent.com.
  3. Run a forward DNS lookup on that hostname and check that it resolves to the same IP address.

The forward lookup is needed because whoever controls an IP block can set any PTR hostname (editorial explanation). Only the real domain’s DNS records confirm the match. This pairing is called forward-confirmed reverse DNS.

As an alternative, Google publishes its crawler IP ranges as JSON files in CIDR format, which can be compared with log addresses.

IP addresses in log files can be personal data (CJEU, judgment of 19 October 2016, C-582/14, Breyer). This is not legal advice.

Is it really Googlebot?Five-step flow: take the IP address from the log file, run a reverse DNS lookup, check that the hostname ends in googlebot.com, google.com or googleusercontent.com, run a forward DNS lookup on that hostname, check that the same IP comes back. Only if both checks pass does the request come from Google. Alternatively, the IP is matched against the JSON lists Google publishes.1 · IP address from the log file66.249.66.12 · Reverse DNS lookupcrawl-66-249-66-1.googlebot.com3 · Hostname ends ingooglebot.com, google.com, googleusercontent.com?yes4 · Forward DNS lookup on the hostname66.249.66.15 · Same IP as in the log file?yesRequest comes from Googlenonot Googlenonot GoogleOr automaticallyMatch the IP against Google’sJSON lists (CIDR format):common-crawlers.jsonspecial-crawlers.jsonuser-triggered-fetchers.jsonuser-triggered-fetchers-google.jsonuser-triggered-agents.jsonOther crawlers often spoofthe Googlebot user agent.It proves nothing on its own.
Both DNS lookups have to match. For large-scale lookups, Google recommends matching against its published IP lists automatically. Sample values from Google’s documentation. Sources: Google Crawling: Verify requests from Google crawlers and fetchers; Google Search Central: What is Googlebot.

Example

Invented values for illustration:

Shell
$ host 192.0.2.10
10.2.0.192.in-addr.arpa domain name pointer crawl-192-0-2-10.googlebot.com.

$ host crawl-192-0-2-10.googlebot.com
crawl-192-0-2-10.googlebot.com has address 192.0.2.10

Both results match, so the request would count as verified. If the hostname were on another domain, or the forward lookup returned a different address, the request would not be Googlebot.

Common mistakes

  • Checking only the user agent. Anyone can claim to be Googlebot in the user agent string. Google describes the check through the accessing IP address.
  • Skipping the way back. Google lists the forward DNS lookup and the comparison with the original IP address (steps 3 and 4) as part of the procedure.
  • Checking every request by hand. For automated verification, Google offers JSON files with its IP ranges in CIDR format.

Sources

  1. Verifying Googlebot and other Google crawlers – Google Search Centraldevelopers.google.com
  2. DNS reverse lookups – Microsoft Learnlearn.microsoft.com
  3. Reverse DNS lookup – Wikipediaen.wikipedia.org
  4. Domain Names, RFC 1035rfc-editor.org
  5. CJEU, judgment of 19 October 2016, C-582/14 (Breyer) – EUR-Lexeur-lex.europa.eu

Last updated: