Deutsch
Crawling & crawl budget

Crawl budget

Crawl budget is the amount of time and resources Google’s crawlers spend on a site. Google defines it through two factors, the crawl capacity limit and crawl demand, and it determines how many URLs from a site get fetched in a given period. Google treats each hostname as a separate site, so www.example.com and shop.example.com have separate crawl budgets.

At a glance

Type
time and resources Google spends crawling a site
Components
crawl capacity limit and crawl demand
Relevant for (Google's estimate)
1M+ pages changing weekly, or 10,000+ pages changing daily
Visible in
Search Console Crawl Stats report, server log files
Docs
Google: Crawl budget management

Last checked:

What determines crawl budget?

The crawl capacity limit (also called hostload) protects the server. It limits the total time a server spends holding connections open for Google, based on the number of parallel connections and their duration. Every site starts with the same conservative default. When a site responds quickly and reliably, the limit goes up. Slower responses, server errors (5xx), or rate-limiting signals such as HTTP 429 push it down.

Crawl demand is how much a crawler wants to fetch from a site. Google notes that each crawler has its own demand. For Google Search it depends on how many URLs Google knows, how popular they are, and how often their content changes. Large numbers of duplicate or low-value URLs mean fewer fetches go to the pages that matter.

Not every crawled page gets indexed. After crawling, Google still evaluates and consolidates each page before deciding whether to add it to the index.

Crawl budget from crawl capacity and crawl demandTwo factors determine crawl budget. Crawl capacity is how much Google can crawl without overloading the server; it rises with stable, fast responses and drops with slow responses, 5xx errors and 429. Crawl demand is how much Google wants to crawl; it depends on the known URL inventory, popularity and staleness. Below are URL types that use up budget without benefit: duplicates, faceted URLs, soft 404s, long redirect chains and sorted variants.Crawl capacityHow much Google can crawlwithout overloading the server↑ stable, fast responses↓ slow responses, 5xx, 429Crawl demandHow much Google wants to crawl• perceived URL inventory• popularity of the URLs• staleness: picking up changesCrawl budgetthe URLs Google canand wants to crawlschematic, not a formulaUSES UP BUDGET WITHOUT BENEFITDuplicatesFacetedURLsSoft 404sLong redirectchainsSortedvariantsIf too much time goes into URLs like these, Google may not explore the rest of the site.
Google defines crawl budget as the set of URLs that Google can and wants to crawl. According to Google, the perceived URL inventory is the factor a site can control the most. Sources: Google Crawling: Optimize your crawl budget; Google Crawling: Managing crawling of faceted navigation URLs.

Which sites need to care about crawl budget?

Google’s guide is written for large sites with more than a million unique pages whose content changes about weekly, for medium sites with more than 10,000 pages that change daily, and for sites where many URLs show as “Discovered - currently not indexed” in Search Console. Google calls these rough estimates. For other sites, Google says keeping the sitemap up to date and checking the Page Indexing report regularly is enough.

Example

An online store has 40,000 product pages. Its faceted navigation turns color, size, and sort order into several hundred thousand parameter URLs. Googlebot spends a large share of its requests on these variants, and new products take days to show up in search. Blocking the filter URLs in robots.txt frees capacity for the actual product pages.

Common mistakes

  • Using noindex to save crawl budget. Google still requests the page and drops it after seeing the directive, so the fetch is spent anyway.
  • Long redirect chains. Google says they have a negative effect on crawling.
  • Soft 404s. Removed pages that return status 200 keep getting crawled. Google recommends 404 or 410 for pages that are gone for good.
  • Toggling robots.txt to shift budget. Google advises against blocking pages temporarily to push crawling elsewhere. Freed budget only goes to other pages if Google is already hitting the site’s capacity limit.

The Crawl Stats report in Search Console and server log files show how many requests Googlebot actually makes.

FAQ

How can you increase crawl budget? Google names two levers. If the server is the bottleneck, for example when the URL Inspection tool reports “Hostload exceeded”, add server resources. To raise crawl demand, improve the quality and usefulness of the content.

How do you check your crawl budget? Google does not report a single crawl budget number. The Crawl Stats report and server logs, mentioned above, are the practical way to see it.

Sources

  1. Crawl budget management – Google Crawling Infrastructure (last updated 2026-07-22)developers.google.com
  2. Crawl Stats report – Search Console Helpsupport.google.com

Last updated: