Deutsch
Log data & formats

Log file

Also known as: Server-Log

A log file is a record that a system writes automatically while it runs. For SEO, the access log of a web server matters most. It records every request the server processed, including requests from search engine crawlers.

How are log files structured?

A web server such as Apache keeps two main logs. The error log stores diagnostic messages and errors. The access log stores each request. Old logs are typically rotated, for example every 24 hours or when a file reaches a set size.

The Common Log Format lists the client IP address, an identity field, the authenticated user, the timestamp, the request line (method, URL and protocol), the status code and the response size. The Combined Log Format adds the referrer and the user agent. These two fields make it useful for SEO, because the user agent shows which crawler or browser asked.

IP addresses can be personal data (CJEU, judgment of 19 October 2016, C-582/14, Breyer).

This is not legal advice.

One log line in Combined Log Format, field by fieldSample line from an access log in Combined Log Format, split into IP address, identity and user, timestamp, request line, status code, bytes, referrer and user agent. The user agent claims to be Googlebot, but the IP address comes from a range reserved for documentation. Only a DNS or IP list check shows whether Google made the request.access.log, one line (wrapped)203.0.113.7 - - [08/Oct/2026:10:15:32 +0200] "GET /en/glossary/ HTTP/1.1" 200 5120 "-""Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"IP address203.0.113.7client, or the proxy in betweenIdentity, user- -“-” means not availableTimestamp[08/Oct/2026:10:15:32 +0200]when the request arrivedRequest"GET /en/glossary/ HTTP/1.1"method, path, protocolStatus200status code of the responseBytes5120size without headers, “-” if emptyReferrer"-"referring page, none hereUser agentline 2what the client says about itselfThe user agent claims to be Googlebot, but the IP is in a range reserved for documentation (RFC 5737).Only a DNS or IP list check shows whether Google made the request.
Apache and nginx write the same fields in the same order in the “combined” format. IP addresses can be personal data (CJEU, judgment of 19 October 2016, C-582/14). Not legal advice. Sources: Apache HTTP Server: Log Files (Combined Log Format); nginx: Module ngx_http_log_module (log_format combined); RFC 5737: IPv4 Address Blocks Reserved for Documentation; CJEU: Press release 112/16 on judgment C-582/14.

Example

A line in Combined Log Format (sample data):

Text
192.0.2.10 - - [06/Oct/2026:09:14:22 +0000] "GET /blog/ HTTP/1.1" 200 18452 "-" "ExampleBot/1.0"

The client at 192.0.2.10 requested /blog/ with GET and received status 200 and 18,452 bytes. It calls itself “ExampleBot/1.0”. No referrer was sent, so the field shows a hyphen.

What can be read from it?

Logs show which URLs a crawler actually requested, how often and with which status code. The user agent alone does not prove who sent a request, because anyone can set it.

Apache also warns that logs can contain unescaped data supplied by clients, so log files should be handled with care.

Common mistakes

  • Not rotating logs. According to Apache, the access log on a moderately busy server grows by about 1 MB or more per 10,000 requests. Without rotation, logs use disk space indefinitely.
  • Forgetting the signal after rotation. With external tools such as logrotate, httpd has to reopen its files (Apache names a graceful restart). Otherwise the server keeps writing to the old file.
  • Processing raw lines without care. Clients can put unescaped content into log lines, including control characters. Apache advises care when handling raw logs.
  • Enabling HostnameLookups. The server then logs host names instead of IP addresses. Apache does not recommend it because it can slow the server significantly.
  • Reading the access log as all traffic. It records requests the server processed. A request answered by an upstream cache or CDN does not appear there (a conclusion, not a statement of the Apache documentation).

FAQ

What is in a log file? Timestamped entries about events. In a web server access log, each entry describes one request.

What opens a .log file? Any text editor, since most logs are plain text.

Sources

  1. Log Files – Apache HTTP Server Documentationhttpd.apache.org
  2. CJEU, judgment of 19 October 2016, C-582/14 (Breyer) – EUR-Lexeur-lex.europa.eu
  3. Log files: definition, types and uses – Sumo Logic (secondary)sumologic.com

Last updated: