Deutsch
Indexing & index status

Crawlability

Crawlability describes whether a crawler such as Googlebot can reach and fetch a URL. It is the precondition for a page to enter a search index at all. Google does not guarantee that it crawls every page, even when a page is technically reachable.

What does crawlability depend on?

Google finds pages through links on known pages and through submitted sitemaps. A crawlable link is an <a> element with an href attribute. Links that only work through script events cannot be reliably extracted. Important pages need at least one link from another page on the site.

Access can be blocked by robots.txt rules, logins or server problems. If a server returns errors or answers slowly, Googlebot lowers its crawling speed. If robots.txt cannot be fetched properly, Google may slow or stop crawling.

Crawlability does not tell whether the page ends up in the index. That depends on indexability, for example noindex or the canonical.

From URL to search resultFive stations from left to right: discovery via links and sitemaps, crawling after checking robots.txt, rendering after waiting in the render queue, indexing with duplicate clustering and canonical selection, serving matching results. Links found in the rendered HTML lead back to discovery. Below each station are typical hurdles according to Google.1 · CRAWLING2 · INDEXING3 · SERVINGDiscoverlinks onknown pages,sitemapsCrawlcheck robots.txt,then fetchthe URLRenderrender queueChromium runsthe JavaScriptIndexcluster pages,pick canonical,storeServeresults fromthe index forthe querynew links from the rendered HTMLTYPICAL HURDLES ACCORDING TO GOOGLEno link,no sitemap*server ornetwork errors,robots.txt blocksno 200 statusor noindex: mayskip renderinglow quality,noindex, duplicateof another URLnot relevant,low quality,robots meta ruleSchematic. Google counts discovery and rendering as part of the crawling stage.* An inference, not a Google statement.
Schematic path of a URL through Google Search. Google describes three stages: crawling (including discovery and rendering), indexing and serving search results. Indexing is not guaranteed. Sources: Google Search Central: In-depth guide to how Google Search works; Google Search Central: Understand JavaScript SEO basics.

Example

A menu opens categories through a JavaScript click handler with no href in the HTML. The category pages are then only discoverable through the sitemap.

HTML
<a href="/shoes/">Shoes</a>
<span onclick="goto('/shoes/')">Shoes</span>

Google can parse the first link and cannot reliably use the second.

Common mistakes

  • a Disallow rule covering a whole directory tree
  • pages with no internal links
  • content behind a login

FAQ

How do I check crawlability? Check robots.txt, internal links and server responses. Search Console’s Crawl Stats report shows host availability for Googlebot.

Sources

  1. In-depth guide to how Google Search works – Google Search Centraldevelopers.google.com
  2. Link best practices for Google – Google Search Centraldevelopers.google.com
  3. Crawl Stats report – Search Console Helpsupport.google.com

Last updated: