# Crawl rate

Category: [Crawling & Indexing](https://seo-geo-glossary.com/en/glossary/category/crawling-indexing/) › [Crawling & crawl budget](https://seo-geo-glossary.com/en/glossary/category/crawling-indexing/crawling-crawl-budget/) · Also known as: crawl speed

**Crawl rate** is the number of requests a crawler sends to a website within a given period of time. For Googlebot, it results from how much load the server can handle and how much Google wants to crawl.

## How does Google set the crawl rate?

Google's goal is to fetch as many pages as possible on each visit without overloading the server. The upper bound is the crawl capacity limit. It rises when the server answers quickly and consistently, and it falls when response times grow or the server returns 5xx errors or HTTP 429. How much Google wants to crawl inside that limit is called crawl demand. Google's documentation does not define “crawl rate” as a term of its own. The description here summarizes its crawl budget and reduce-crawl-rate pages.

Search Console no longer has a setting for it. Google announced in November 2023 that it would remove the crawl rate limiter tool because automatic handling had improved. Press coverage dates the removal to January 8, 2024. Google also says you cannot request a higher crawl rate.

## How can the crawl rate be reduced?

For emergencies, Google recommends returning `500`, `503` or `429` instead of `200` for a couple of hours up to two days. Googlebot then lowers its rate across the whole hostname and raises it again once the errors stop. A `Retry-After` header can suggest when to come back. Longer periods can lead to URLs being dropped from the index, and during the slowdown new pages are discovered later.

The robots.txt field `crawl-delay` does not change Google's crawl rate, because Google does not support it.

## Example

A shop moves to new servers and sees load spikes from Googlebot. For a few hours it answers crawl requests with `503 Service Unavailable` and `Retry-After: 3600`. Once the move is done, it serves `200` again and Google raises the rate on its own.

Crawl speed is used as an informal synonym. Google's documentation says “crawl rate”.

*Diagram: Crawl budget from capacity and demand.* Google defines crawl budget as the set of URLs that Google can and wants to crawl. According to Google, the perceived URL inventory is the factor a site can control the most.

## Not to be confused with

- [Crawl demand](https://seo-geo-glossary.com/en/glossary/crawl-demand/index.md): describes how much a crawler wants to crawl on a site. Google names three factors: perceived inventory, popularity of the URLs and staleness.

## Related terms

- [Crawl budget](https://seo-geo-glossary.com/en/glossary/crawl-budget/index.md)
- [Crawl demand](https://seo-geo-glossary.com/en/glossary/crawl-demand/index.md)
- [Crawl depth](https://seo-geo-glossary.com/en/glossary/crawl-depth/index.md)
- [Crawl-delay](https://seo-geo-glossary.com/en/glossary/crawl-delay/index.md)
- [503 Service Unavailable](https://seo-geo-glossary.com/en/glossary/503-service-unavailable/index.md)
- [429 Too Many Requests](https://seo-geo-glossary.com/en/glossary/429-too-many-requests/index.md)
- [Crawl stats report](https://seo-geo-glossary.com/en/glossary/crawl-stats-report/index.md)
- [Crawl frequency](https://seo-geo-glossary.com/en/glossary/crawl-frequency/index.md)

## Sources

- [Reduce the Google crawl rate – Google Crawling Infrastructure](https://developers.google.com/crawling/docs/crawlers-fetchers/reduce-crawl-rate)
- [Optimize your crawl budget – Google Crawling Infrastructure](https://developers.google.com/crawling/docs/crawl-budget)
- [Upcoming deprecation of Crawl Rate Limiter Tool in Search Console – Google Search Central Blog (11/2023)](https://developers.google.com/search/blog/2023/11/sc-crawl-limiter-byebye)
- [Robots.txt specification – Google Search Central](https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt)
- [Googlebot crawl rate tool is now gone – Search Engine Land (01/2024)](https://searchengineland.com/googlebot-crawl-rate-tool-is-now-gone-436382)

---

Last updated: October 8, 2026
URL: https://seo-geo-glossary.com/en/glossary/crawl-rate/