Crawl-delay is a robots.txt directive that asks a crawler to wait between requests. It is not part of the core robots.txt standard, and each crawler operator decides whether and how to apply it.
How is crawl-delay handled?
Google supports only the fields user-agent, allow, disallow and sitemap. Other fields such as crawl-delay are ignored. RFC 9309, the IETF standard for robots.txt, does not define the directive either.
According to a 2012 post on the Bing Webmaster Blog, Bing reads it and the value is not a crawl rate. It sets the size of a time window of 1 to 30 seconds in which BingBot fetches at most one page. With a delay of 5, that works out to roughly 17,280 pages per day at most. A higher value means fewer crawled pages, which Bing says can lead to less content being indexed.
To slow Googlebot down, Google recommends HTTP status codes instead. For short emergencies, 500, 503 or 429 work, optionally with a Retry-After header. Google warns against doing this for more than one to two days.
Example
User-agent: bingbot
Crawl-delay: 5
The rule is read by Bing. Googlebot ignores it.
FAQ
Does Google respect crawl-delay? No. Google lists it among the fields it does not support.
Sources
- robots.txt specification – Google Search Centraldevelopers.google.com
- Reduce the Google crawl rate – Google Crawling Infrastructuredevelopers.google.com
- To crawl or not to crawl, that is BingBot's question – Bing Webmaster Blog (05/2012)blogs.bing.com
- RFC 9309: Robots Exclusion Protocol – IETFrfc-editor.org
Last updated: