CDN (content delivery network)

How does a CDN work?

A CDN operates many points of presence (PoPs). The servers at these locations are called edge servers. When a request arrives, the CDN routes it to a suitable edge server. If the edge server holds a current copy, it answers directly. If not, it fetches the file from the origin, passes it on and usually stores it for later requests. Besides caching, CDNs handle load balancing and request routing.

MDN names the benefits as less load on the origin server, lower latency through shorter distances, preconfigured caching for static files and the ability to handle traffic peaks. It also lists drawbacks: a dependency on a third party, additional attack surface and, for files shared across sites, extra DNS lookups. Modern browsers no longer share cached files between websites for privacy reasons, so a shared library URL gives no caching advantage across sites.

By default, a CDN such as Cloudflare caches static files like images, CSS and JavaScript, but not HTML pages. In a case study by searchVIU (2019), an online shop also turned on HTML caching, except for logged-in visitors and those with items in their cart. Pages stayed cached for 30 minutes to match how often prices were updated. Average page load time fell by 28%, from 4.49 to 3.25 seconds.

What Google says about CDNs and crawling

According to Google’s Search Central blog (December 2024), Google’s crawling infrastructure allows higher crawl rates for sites backed by a CDN, which it infers from the IP address serving the URLs. The CDN’s cache is empty when a URL is first requested, so the origin server still has to deliver every URL at least once. Google therefore expects the crawl rate to stay high for a few days when many URLs launch at once.

CDN bot protection can block crawlers. For a temporary block, Google prefers HTTP status 503 or 429. Network timeouts can lead to affected URLs being removed from the index, and error messages served with status 200 can cause pages to be treated as duplicates. Google supports both a separate hostname such as cdn.example.com for static files and a main hostname backed by a CDN.

What to check for SEO

Requests from crawlers reach the CDN first, so the status codes, headers and cached versions they receive depend on the CDN configuration. Points worth checking:

• Cache rules for HTML pages: a stale cached page shows old content or old redirects.
• Cache-Control and s-maxage, which set how long shared caches such as CDNs keep a response.
• Whether security or bot rules on the CDN block or throttle legitimate crawlers.
• Whether the CDN delivers the same URLs and canonical tags as the origin.

Example

Cache-Control: public, s-maxage=604800

This lets shared caches such as a CDN store the response for seven days.

Last updated: October 8, 2026