Crawl errors

What types of crawl errors are there?

Two levels can be told apart. Site-level errors affect all URLs of a host. In Google’s Crawl Stats report, the host status covers three categories for the last 90 days: robots.txt fetching, DNS resolution and server connectivity. If Google gets neither a valid robots.txt nor a 404-type answer (a 403, 404 or 410 counts as success), it slows or stops crawling until the file is reachable again. The report is only available for root-level properties.

URL-level errors affect single URLs. The page indexing report lists reasons such as Server error (5xx), Not found (404), Soft 404, Redirect error and URL blocked by robots.txt. Not every one needs a fix. A URL blocked by an intended robots.txt rule or a removed page with a 404 is expected behavior.

Persistent server errors also lower the crawl capacity limit, so Google fetches fewer pages overall.

Until 2019, Search Console had a dedicated Crawl Errors report. Older guides, including a post by searchVIU (2018), still told readers to mark the list as fixed at regular intervals. Google retired the report and its API in 2019 and has since pointed site owners to the Page indexing report (formerly Index Coverage) and the URL Inspection tool.

Example

A server returns 503 for /robots.txt during a deployment. The Crawl Stats report shows robots.txt fetching as a problem and Google slows crawling of the whole host until the file loads again.

FAQ

How do I fix crawl errors? First identify the level. Host status problems point to DNS, server or robots.txt. For single URLs, the URL Inspection tool shows what Googlebot received.

Last updated: October 8, 2026