Deutsch
Crawlers & bots

User agent

Also known as: User-Agent header, UA string

The user agent is the software that makes a request to a web server on behalf of a user, and the name of its HTTP request header. In the header, the client names itself, for example as a browser, a crawler or a command-line tool. Servers use it to decide how to respond, and log files record it, which makes it the standard way to tell Googlebot from a normal browser visit.

How is the header structured?

The value consists of product tokens with optional comments in parentheses. Browsers begin with Mozilla/5.0 for historical reasons and then add system and engine details. Crawlers add their own name. Examples (the Chrome version number changes over time):

HTTP
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/143.0.0.0 Safari/537.36

Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

The second line is Google’s smartphone crawler, where W.X.Y.Z stands for the current Chrome version. Modern browsers also shorten the string, for instance by freezing the platform version, to limit fingerprinting. Detailed device data is then available through User-Agent Client Hints.

Why does it matter for SEO?

  • robots.txt. A group starts with User-agent: Googlebot. The value there is the crawler’s token, which is shorter than the full header string.
  • Log files. Requests can be grouped by crawler to see which bots visit and how often.
  • Testing. Crawlers and browser extensions can send a different user agent to see what a server returns to a crawler.
  • Different content per client. MDN advises against switching content by user agent because detection is unreliable. For search crawlers, showing them different content than users is cloaking.
One log line in Combined Log Format, field by fieldSample line from an access log in Combined Log Format, split into IP address, identity and user, timestamp, request line, status code, bytes, referrer and user agent. The user agent claims to be Googlebot, but the IP address comes from a range reserved for documentation. Only a DNS or IP list check shows whether Google made the request.access.log, one line (wrapped)203.0.113.7 - - [08/Oct/2026:10:15:32 +0200] "GET /en/glossary/ HTTP/1.1" 200 5120 "-""Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"IP address203.0.113.7client, or the proxy in betweenIdentity, user- -“-” means not availableTimestamp[08/Oct/2026:10:15:32 +0200]when the request arrivedRequest"GET /en/glossary/ HTTP/1.1"method, path, protocolStatus200status code of the responseBytes5120size without headers, “-” if emptyReferrer"-"referring page, none hereUser agentline 2what the client says about itselfThe user agent claims to be Googlebot, but the IP is in a range reserved for documentation (RFC 5737).Only a DNS or IP list check shows whether Google made the request.
Apache and nginx write the same fields in the same order in the “combined” format. IP addresses can be personal data (CJEU, judgment of 19 October 2016, C-582/14). Not legal advice. Sources: Apache HTTP Server: Log Files (Combined Log Format); nginx: Module ngx_http_log_module (log_format combined); RFC 5737: IPv4 Address Blocks Reserved for Documentation; CJEU: Press release 112/16 on judgment C-582/14.

Can the user agent be faked?

Yes. Any client can send any string, and Google states that the user agent can be spoofed. A request that claims to be Googlebot can be verified through a reverse DNS lookup followed by a forward lookup, or by checking the IP address against the ranges Google publishes.

Sources

  1. User-Agent header – MDN Web Docsdeveloper.mozilla.org
  2. RFC 9110: HTTP Semantics, section 10.1.5 User-Agentrfc-editor.org
  3. Google's common crawlers – Google Search Centraldevelopers.google.com
  4. Verifying Googlebot and other Google crawlers – Google Search Centraldevelopers.google.com
  5. User agent – Wikipedia (secondary)en.wikipedia.org

Last updated: