Deutsch
Indexing & index status

Indexability

Indexability describes whether a URL is technically allowed to be added to a search index. It depends on signals such as the HTTP status code, noindex rules and, in many SEO tools, the canonical. Whether a search engine then indexes the page is its own decision, and Google says it does not guarantee indexing.

Which signals decide indexability?

Google’s documentation does not use the term as a category, while SEO tools do. Tools treat a URL as indexable when nothing prevents indexing. Typical checks are:

  • the URL returns a success status instead of an error or redirect
  • no noindex in the robots meta tag or in the X-Robots-Tag header
  • no canonical pointing to a different URL (tool practice, for example in Sitebulb)

noindex has a precondition. Google must be allowed to crawl the page, because it only reads the rule when it fetches the page. If robots.txt blocks the URL, the rule is never seen.

From URL to search resultFive stations from left to right: discovery via links and sitemaps, crawling after checking robots.txt, rendering after waiting in the render queue, indexing with duplicate clustering and canonical selection, serving matching results. Links found in the rendered HTML lead back to discovery. Below each station are typical hurdles according to Google.1 · CRAWLING2 · INDEXING3 · SERVINGDiscoverlinks onknown pages,sitemapsCrawlcheck robots.txt,then fetchthe URLRenderrender queueChromium runsthe JavaScriptIndexcluster pages,pick canonical,storeServeresults fromthe index forthe querynew links from the rendered HTMLTYPICAL HURDLES ACCORDING TO GOOGLEno link,no sitemap*server ornetwork errors,robots.txt blocksno 200 statusor noindex: mayskip renderinglow quality,noindex, duplicateof another URLnot relevant,low quality,robots meta ruleSchematic. Google counts discovery and rendering as part of the crawling stage.* An inference, not a Google statement.
Schematic path of a URL through Google Search. Google describes three stages: crawling (including discovery and rendering), indexing and serving search results. Indexing is not guaranteed. Sources: Google Search Central: In-depth guide to how Google Search works; Google Search Central: Understand JavaScript SEO basics.

How does it differ from crawlability?

Crawlability asks whether a crawler can reach and fetch a URL. Indexability asks whether the fetched URL may enter the index. A page can be crawlable and not indexable, for example with noindex.

Example

HTML
<meta name="robots" content="noindex">

This tag makes the URL non-indexable for search engines that support the rule, such as Google.

FAQ

How do you stop Google from indexing a page? Use noindex in a meta tag or HTTP header and keep the URL crawlable so Google can read it.

Sources

  1. Block Search indexing with noindex – Google Search Centraldevelopers.google.com
  2. In-depth guide to how Google Search works – Google Search Centraldevelopers.google.com
  3. Robots meta tag, data-nosnippet, and X-Robots-Tag specifications – Google Search Centraldevelopers.google.com
  4. Indexability hints – Sitebulbsitebulb.com

Last updated: