# Crawl-delay

Category: [Crawling & Indexing](https://seo-geo-glossary.com/en/glossary/category/crawling-indexing/) › [robots.txt & robots directives](https://seo-geo-glossary.com/en/glossary/category/crawling-indexing/robots-txt-directives/)

**Crawl-delay** is a robots.txt directive that asks a crawler to wait between requests. It is not part of the core robots.txt standard, and each crawler operator decides whether and how to apply it.

## How is crawl-delay handled?

Google supports only the fields `user-agent`, `allow`, `disallow` and `sitemap`. Other fields such as `crawl-delay` are ignored. RFC 9309, the IETF standard for robots.txt, does not define the directive either.

According to a 2012 post on the Bing Webmaster Blog, Bing reads it and the value is not a crawl rate. It sets the size of a time window of 1 to 30 seconds in which BingBot fetches at most one page. With a delay of 5, that works out to roughly 17,280 pages per day at most. A higher value means fewer crawled pages, which Bing says can lead to less content being indexed.

To slow Googlebot down, Google recommends HTTP status codes instead. For short emergencies, `500`, `503` or `429` work, optionally with a `Retry-After` header. Google warns against doing this for more than one to two days.

## Example

```
User-agent: bingbot
Crawl-delay: 5
```

The rule is read by Bing. Googlebot ignores it.

## FAQ

**Does Google respect crawl-delay?**
No. Google lists it among the fields it does not support.

## Related terms

- [robots.txt](https://seo-geo-glossary.com/en/glossary/robots-txt/index.md)
- [Crawl rate](https://seo-geo-glossary.com/en/glossary/crawl-rate/index.md)
- [Crawl budget](https://seo-geo-glossary.com/en/glossary/crawl-budget/index.md)
- [Crawl demand](https://seo-geo-glossary.com/en/glossary/crawl-demand/index.md)
- [429 Too Many Requests](https://seo-geo-glossary.com/en/glossary/429-too-many-requests/index.md)

## Sources

- [robots.txt specification – Google Search Central](https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt)
- [Reduce the Google crawl rate – Google Crawling Infrastructure](https://developers.google.com/crawling/docs/crawlers-fetchers/reduce-crawl-rate)
- [To crawl or not to crawl, that is BingBot's question – Bing Webmaster Blog (05/2012)](https://blogs.bing.com/webmaster/2012/05/03/to-crawl-or-not-to-crawl-that-is-bingbots-question)
- [RFC 9309: Robots Exclusion Protocol – IETF](https://www.rfc-editor.org/rfc/rfc9309.html)

---

Last updated: October 7, 2026
URL: https://seo-geo-glossary.com/en/glossary/crawl-delay/