Crawl frequency

What determines crawl frequency?

Google describes crawl budget as the set of URLs it can and wants to crawl, and splits it into two parts. The crawl capacity limit rises when a site responds consistently and quickly. It drops when the server returns 5xx errors or rate-limiting signals such as HTTP 429. Crawl demand depends on factors such as how popular URLs are and how stale Google’s copy is. Google also states that when demand is low, it crawls a site less, even if the capacity limit has not been reached.

Slowing Googlebot down

To reduce crawling urgently, Google recommends returning 500, 503 or 429 status codes for a short period, such as a couple of hours or one to two days. A longer period can have a negative effect on how a site appears in Google products. Google announced the retirement of the Crawl Rate Limiter tool in Search Console in November 2023. Press coverage dates its removal to January 8, 2024.

Where to see it

The Crawl Stats report in Search Console shows total crawl requests, download size, average response time and host status over the past 90 days. Google says sites with fewer than a thousand pages should not need it. Server logs additionally show requests from other bots.

Crawl frequency, crawl rate and crawl budget

The three terms are often used interchangeably. Google only defines crawl budget, as the set of URLs that Google can and wants to crawl. In this glossary, crawl frequency means how often a URL or site is requested, and crawl rate means the number of requests within a period of time (assessment).

Example

A shop launches a new filter that creates thousands of parameter URLs. Crawl Stats shows a jump in requests, while new product pages take longer to appear in the index. Excluding the filter URLs from crawling shifts requests back to the pages that matter.

FAQ

How often does Google crawl a website? There is no fixed schedule. Frequency differs by site and by URL.

Last updated: October 7, 2026