HTTP compression is the process of shrinking text files such as HTML, CSS and JavaScript on the server before they travel over the network. The browser or crawler unpacks them on arrival. The two common formats are gzip and Brotli.
How does the negotiation work?
The client lists the formats it can unpack in the Accept-Encoding request header. The server picks one, compresses the response and names the choice in the Content-Encoding response header. Vary: Accept-Encoding tells caches that the same URL can exist in several variants.
GET /styles.css HTTP/1.1
Accept-Encoding: br, gzip
HTTP/1.1 200 OK
Content-Encoding: br
Vary: Accept-Encoding
MDN lists several values for Content-Encoding, among them gzip, deflate, br and zstd. gzip is the most common. br stands for Brotli, which MDN describes as having better compression ratios than gzip, though it compresses more slowly. zstd (Zstandard) is newer still, and deflate is older.
Brotli was developed by Google and is specified in RFC 7932. It is a lossless algorithm, which means the unpacked file is identical to the original.
Does compression matter for crawlers?
Google states that its crawlers and fetchers support gzip, deflate and Brotli (br). A page compressed with one of these formats stays readable for Googlebot. Whether compression affects rankings is not documented, so this entry makes no claim about it. The effect is on file size and transfer time.
What should be compressed?
Text formats compress well, so HTML, CSS, JavaScript, JSON and SVG are the usual candidates. MDN advises against compressing formats that are already compressed, such as JPEG, PNG, audio and video, because the extra work costs more than it saves.
How to check it
Open the browser’s developer tools, switch to the Network tab and select a text file. If the response headers contain no Content-Encoding line, the file was sent uncompressed. Compression is switched on in the server configuration or at the host. MDN names mod_deflate for Apache and ngx_http_gzip_module for Nginx.
Common mistakes
- No
Vary: Accept-Encoding. MDN says the server must send at leastAccept-Encodingin theVaryheader when negotiating compression, so caches can store the variants separately. - Compressing already compressed files again. For JPEG, PNG, WebP, audio and video, MDN says the overhead can exceed the gain, and the file can even grow.
- Text files left uncompressed. MDN recommends activating compression for all files except already compressed ones. A missing
Content-Encodingon HTML, CSS or JavaScript points to a gap.
FAQ
What is Brotli used for? Brotli compresses web assets for transfer. Browsers announce support with br in Accept-Encoding.
Is Brotli better than gzip? MDN says Brotli achieves better compression ratios but compresses more slowly than gzip. Gzip is the most common format.
Sources
- HTTP compression – MDN Web Docsdeveloper.mozilla.org
- Overview of Google crawlers and fetchers – Google Search Centraldevelopers.google.com
- Brotli compression – MDN Web Docs Glossarydeveloper.mozilla.org
- google/brotli README – GitHubgithub.com
- HTTP Semantics (RFC 9110), Content Codings – IETFrfc-editor.org
Last updated: