Deutsch
Crawling & crawl budget

URL discovery

Also known as: URL discovery

URL discovery is the step in which a search engine learns that a URL exists, before it fetches the page. In SEO, the term describes how Google finds new addresses on a website. Google’s own documentation calls this process “URL discovery”.

How does Google discover URLs?

According to Google, there is no central registry of all web pages, so it has to look for new and updated pages continuously. It finds URLs in three ways: by revisiting pages it already knows, by following links from known pages, and by processing sitemaps that site owners submit.

Links are the main route. Google says every page you care about should have a link from at least one other page on your site. A page without internal links, an orphan page, can therefore only be found through a sitemap or an external link.

A test by searchVIU (2017) with a control page found that an internal link injected with JavaScript was enough for Google to find and index a page. In the same test, Google also indexed a URL that was mentioned only inside a JSON-LD block. Google doesn’t document this as a discovery path, so it is best not to rely on it.

From URL to search resultFive stations from left to right: discovery via links and sitemaps, crawling after checking robots.txt, rendering after waiting in the render queue, indexing with duplicate clustering and canonical selection, serving matching results. Links found in the rendered HTML lead back to discovery. Below each station are typical hurdles according to Google.1 · CRAWLING2 · INDEXING3 · SERVINGDiscoverlinks onknown pages,sitemapsCrawlcheck robots.txt,then fetchthe URLRenderrender queueChromium runsthe JavaScriptIndexcluster pages,pick canonical,storeServeresults fromthe index forthe querynew links from the rendered HTMLTYPICAL HURDLES ACCORDING TO GOOGLEno link,no sitemap*server ornetwork errors,robots.txt blocksno 200 statusor noindex: mayskip renderinglow quality,noindex, duplicateof another URLnot relevant,low quality,robots meta ruleSchematic. Google counts discovery and rendering as part of the crawling stage.* An inference, not a Google statement.
Schematic path of a URL through Google Search. Google describes three stages: crawling (including discovery and rendering), indexing and serving search results. Indexing is not guaranteed. Sources: Google Search Central: In-depth guide to how Google Search works; Google Search Central: Understand JavaScript SEO basics.

Discovered, crawled, indexed

These are separate steps. Googlebot visits discovered URLs to crawl them, and Google states that indexing is not guaranteed. A sitemap helps discovery but does not guarantee that every listed URL is crawled or indexed. Google says sitemaps matter most for large sites of about 500 pages or more and for new sites with few external links.

Example

A new product category is added to the main navigation. The next time Googlebot crawls the homepage, it finds the link. The category URL is now discovered and waits to be crawled.

More on the searchVIU blog

Sources

  1. In-depth guide to how Google Search works – Google Search Centraldevelopers.google.com
  2. Build and submit a sitemap – Google Search Centraldevelopers.google.com
  3. Link best practices for Google – Google Search Centraldevelopers.google.com
  4. More JavaScript SEO experiments with Google Tag Manager – searchVIU blog (2017)searchviu.com
  5. Understand the JavaScript SEO basics – Google Search Centraldevelopers.google.com

Last updated: