Deutsch
Crawling & crawl budget

Crawl rate

Also known as: crawl speed

Crawl rate is the number of requests a crawler sends to a website within a given period of time. For Googlebot, it results from how much load the server can handle and how much Google wants to crawl.

How does Google set the crawl rate?

Google’s goal is to fetch as many pages as possible on each visit without overloading the server. The upper bound is the crawl capacity limit. It rises when the server answers quickly and consistently, and it falls when response times grow or the server returns 5xx errors or HTTP 429. How much Google wants to crawl inside that limit is called crawl demand. Google’s documentation does not define “crawl rate” as a term of its own. The description here summarizes its crawl budget and reduce-crawl-rate pages.

Search Console no longer has a setting for it. Google announced in November 2023 that it would remove the crawl rate limiter tool because automatic handling had improved. Press coverage dates the removal to January 8, 2024. Google also says you cannot request a higher crawl rate.

Crawl budget from crawl capacity and crawl demandTwo factors determine crawl budget. Crawl capacity is how much Google can crawl without overloading the server; it rises with stable, fast responses and drops with slow responses, 5xx errors and 429. Crawl demand is how much Google wants to crawl; it depends on the known URL inventory, popularity and staleness. Below are URL types that use up budget without benefit: duplicates, faceted URLs, soft 404s, long redirect chains and sorted variants.Crawl capacityHow much Google can crawlwithout overloading the server↑ stable, fast responses↓ slow responses, 5xx, 429Crawl demandHow much Google wants to crawl• perceived URL inventory• popularity of the URLs• staleness: picking up changesCrawl budgetthe URLs Google canand wants to crawlschematic, not a formulaUSES UP BUDGET WITHOUT BENEFITDuplicatesFacetedURLsSoft 404sLong redirectchainsSortedvariantsIf too much time goes into URLs like these, Google may not explore the rest of the site.
Google defines crawl budget as the set of URLs that Google can and wants to crawl. According to Google, the perceived URL inventory is the factor a site can control the most. Sources: Google Crawling: Optimize your crawl budget; Google Crawling: Managing crawling of faceted navigation URLs.

How can the crawl rate be reduced?

For emergencies, Google recommends returning 500, 503 or 429 instead of 200 for a couple of hours up to two days. Googlebot then lowers its rate across the whole hostname and raises it again once the errors stop. A Retry-After header can suggest when to come back. Longer periods can lead to URLs being dropped from the index, and during the slowdown new pages are discovered later.

The robots.txt field crawl-delay does not change Google’s crawl rate, because Google does not support it.

Example

A shop moves to new servers and sees load spikes from Googlebot. For a few hours it answers crawl requests with 503 Service Unavailable and Retry-After: 3600. Once the move is done, it serves 200 again and Google raises the rate on its own.

Crawl speed is used as an informal synonym. Google’s documentation says “crawl rate”.

Sources

  1. Reduce the Google crawl rate – Google Crawling Infrastructuredevelopers.google.com
  2. Optimize your crawl budget – Google Crawling Infrastructuredevelopers.google.com
  3. Upcoming deprecation of Crawl Rate Limiter Tool in Search Console – Google Search Central Blog (11/2023)developers.google.com
  4. Robots.txt specification – Google Search Centraldevelopers.google.com
  5. Googlebot crawl rate tool is now gone – Search Engine Land (01/2024)searchengineland.com

Last updated: