Deutsch
Crawling & crawl budget

Crawl demand

Crawl demand is the part of crawl budget that describes how much a crawler wants to crawl a site. The other part is the crawl capacity limit, which describes how much the server can handle. Google defines crawl budget as the set of URLs it can and wants to crawl.

What does crawl demand depend on?

Every crawler has its own demand. AdsBot, for example, has a higher demand when a site runs dynamic ad targets, and Google Shopping has a higher demand for products in merchant feeds. For Googlebot, demand varies with a site’s size, update frequency, page quality and relevance compared to other sites. Google names three factors:

  • Perceived inventory: Without guidance, Google tries to crawl all or most URLs it knows. Many duplicates or unwanted URLs waste crawling time. Google says this is the factor site owners can influence most.
  • Popularity: URLs that are more popular on the internet tend to be crawled more often.
  • Staleness: Google wants to recrawl often enough to pick up changes.

A site move can also raise demand for a while, because Google reprocesses the content under the new URLs. If demand is low, Google crawls less even when the capacity limit has not been reached.

Crawl budget from crawl capacity and crawl demandTwo factors determine crawl budget. Crawl capacity is how much Google can crawl without overloading the server; it rises with stable, fast responses and drops with slow responses, 5xx errors and 429. Crawl demand is how much Google wants to crawl; it depends on the known URL inventory, popularity and staleness. Below are URL types that use up budget without benefit: duplicates, faceted URLs, soft 404s, long redirect chains and sorted variants.Crawl capacityHow much Google can crawlwithout overloading the server↑ stable, fast responses↓ slow responses, 5xx, 429Crawl demandHow much Google wants to crawl• perceived URL inventory• popularity of the URLs• staleness: picking up changesCrawl budgetthe URLs Google canand wants to crawlschematic, not a formulaUSES UP BUDGET WITHOUT BENEFITDuplicatesFacetedURLsSoft 404sLong redirectchainsSortedvariantsIf too much time goes into URLs like these, Google may not explore the rest of the site.
Google defines crawl budget as the set of URLs that Google can and wants to crawl. According to Google, the perceived URL inventory is the factor a site can control the most. Sources: Google Crawling: Optimize your crawl budget; Google Crawling: Managing crawling of faceted navigation URLs.

Example

A shop generates thousands of near-identical filter URLs. Google knows them, so the perceived inventory is large and part of the crawling time goes to the variants. Consolidating or blocking them leaves more time for the pages that matter (illustrative example).

FAQ

Does every site need to worry about it? Google’s guide is aimed at sites with about 1 million or more unique pages that change weekly, or about 10,000 or more pages that change daily. Google calls these numbers a rough estimate.

Sources

  1. Optimize your crawl budget – Google Crawling Infrastructuredevelopers.google.com
  2. Reduce the Google crawl rate – Google Crawling Infrastructuredevelopers.google.com

Last updated: