Deutsch
Log data & formats

Log file analysis

Also known as: SEO log file analysis, server log analysis

Log file analysis in SEO means evaluating a web server’s access logs, which record every request for a URL with its timestamp, status code, and user agent. It shows which pages Googlebot, Bingbot, and AI bots actually request, how often, and what response they get.

At a glance

Type
analysis of a web server's access logs
Data source
access logs from the web server or CDN
Common format
Combined Log Format (Apache, nginx combined)
Bot check
reverse and forward DNS lookup or the operator's IP lists
Docs
Google: Verifying Googlebot and other Google crawlers

Last checked:

What does a log file contain?

Many web servers write access logs in the Combined Log Format. Apache documents it as the Common Log Format plus two fields, Referer and User-Agent, and nginx uses the same layout as its predefined combined format. A single line holds the client IP, the time, the request line with the URL, the status code, the response size, the referrer, and the user agent:

Text
192.0.2.10 - - [06/Oct/2026:08:14:03 +0200] "GET /women/running-shoes/ HTTP/1.1" 200 51234 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

Aggregated over millions of lines, this data shows which sections bots visit often, which they rarely reach, and where they hit errors or redirects. A crawler shows what is reachable, and Search Console shows Google’s summarized view. Logs show the requests that really happened.

IP addresses can be personal data (CJEU, judgment of 19 October 2016, C-582/14, Breyer).

This is not legal advice.

One log line in Combined Log Format, field by fieldSample line from an access log in Combined Log Format, split into IP address, identity and user, timestamp, request line, status code, bytes, referrer and user agent. The user agent claims to be Googlebot, but the IP address comes from a range reserved for documentation. Only a DNS or IP list check shows whether Google made the request.access.log, one line (wrapped)203.0.113.7 - - [08/Oct/2026:10:15:32 +0200] "GET /en/glossary/ HTTP/1.1" 200 5120 "-""Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"IP address203.0.113.7client, or the proxy in betweenIdentity, user- -“-” means not availableTimestamp[08/Oct/2026:10:15:32 +0200]when the request arrivedRequest"GET /en/glossary/ HTTP/1.1"method, path, protocolStatus200status code of the responseBytes5120size without headers, “-” if emptyReferrer"-"referring page, none hereUser agentline 2what the client says about itselfThe user agent claims to be Googlebot, but the IP is in a range reserved for documentation (RFC 5737).Only a DNS or IP list check shows whether Google made the request.
Apache and nginx write the same fields in the same order in the “combined” format. IP addresses can be personal data (CJEU, judgment of 19 October 2016, C-582/14). Not legal advice. Sources: Apache HTTP Server: Log Files (Combined Log Format); nginx: Module ngx_http_log_module (log_format combined); RFC 5737: IPv4 Address Blocks Reserved for Documentation; CJEU: Press release 112/16 on judgment C-582/14.

How do you verify a bot?

Anyone can send a request with a Googlebot user agent. Google describes two checks. One is a reverse DNS lookup of the IP address, which must end in googlebot.com, google.com, or googleusercontent.com, followed by a forward lookup that has to return the same IP. The other is matching the IP against the ranges Google publishes as JSON files. OpenAI also publishes IP lists for its bots, for example for OAI-SearchBot and GPTBot.

When is it worth doing?

Mostly on large sites where crawl budget can become a limit. Google’s crawl budget guide is aimed, as a rough guide, at sites with more than one million unique pages that change weekly, or more than 10,000 pages that change daily. After a site migration, logs also show quickly whether bots still request old URLs and which redirects they receive.

Common mistakes

  • Trusting the user agent: Any client can set any user agent. Google points out that spammers also pose as Google crawlers and recommends the DNS check or its published IP lists.
  • Accepting only googlebot.com: Depending on the crawler, the hostname can also end in google.com or googleusercontent.com. Google keeps separate IP lists for common crawlers, special-case crawlers, and user-triggered fetchers. Checking just one list flags real Google requests as fake.
  • Overlooking server errors and 429s: When a site often answers bots with 5xx or 429, Google’s crawl budget guide says the crawl limit goes down and Google crawls less.
  • Ignoring redirect chains: Several 3xx responses in a row for an old URL point to a chain. Google says long redirect chains have a negative effect on crawling.

FAQ

Where do I find my log files? On the web server. The location depends on the configuration: in Apache it is set by the CustomLog directive, in nginx by access_log (default logs/access.log in combined format). Many hosting providers offer downloads in their control panel. If a CDN sits in front of the server, the origin logs only see part of the traffic, and the CDN’s logs become the main source. Using Cloudflare as the example, a 2026 practitioner article lists the requests that never reach the origin log: blocked or challenged requests, responses served from the edge cache, requests answered by Workers and rate-limited requests. If a firewall rule blocks part of Googlebot’s traffic, the origin log only shows a drop in crawling, while the cause is recorded in the CDN’s security events.

How often should logs be analyzed? There is no fixed rule. Continuous evaluation helps most on sites that change often, while a one-off analysis can answer specific questions such as a migration check.

More on the searchVIU blog

Sources

  1. Log Files – Apache HTTP Server 2.4 Documentation (Combined Log Format)httpd.apache.org
  2. Module ngx_http_log_module – nginx documentationnginx.org
  3. Verifying Googlebot and other Google crawlers – Google Search Centraldevelopers.google.com
  4. Large site owner's guide to managing your crawl budget – Google Search Centraldevelopers.google.com
  5. CJEU, judgment of 19 October 2016, C-582/14 (Breyer) – EUR-Lexeur-lex.europa.eu
  6. Overview of OpenAI Crawlers – OpenAIdevelopers.openai.com
  7. Cloudflare Fundamentals for SEO & GEO – searchVIU blog (2026)searchviu.com
  8. Security Events – Cloudflare WAF docsdevelopers.cloudflare.com

Last updated: