Crawling

How does crawling work?

It starts with URL discovery. There is no central registry of all web pages, so Google keeps a list of known URLs and keeps adding to it. Some pages are known from earlier visits, others turn up as links on known pages, and others come from sitemaps that site owners submit. Googlebot then uses an algorithm to decide which sites to crawl, how often, and how many pages to fetch from each.

For each fetch, the crawler first checks robots.txt, then requests the URL and evaluates the server’s response. If the page returns content, Google renders it with a recent version of Chrome, so that content and links added by JavaScript become visible. New links go back into the queue.

Crawl speed also depends on the server. Google’s crawlers are designed to avoid overloading a site, and they slow down when the server responds with errors such as HTTP 500. Google also notes that it does not crawl every page it discovers, for example pages blocked by robots.txt or pages behind a login.

Crawling vs. indexing

A crawled page is not automatically indexed. During indexing, Google analyzes the content, groups duplicates, and picks a canonical URL. The reverse can also happen: a URL blocked in robots.txt may still be indexed, without its content, if other sites link to it. To keep a page out of the index, use noindex and leave the page crawlable so Google can see the directive.

How to control and monitor crawling

• robots.txt tells crawlers which paths they may fetch. The rules were standardized as RFC 9309 in 2022.
• XML sitemaps list URLs you want crawled.
• Internal links decide which pages a crawler can reach at all.
• Status codes and redirects determine whether a request returns content or ends in an error.

In Google Search Console, the Crawl Stats report shows total requests, download size, response times, host status, response codes, file types, crawl purpose, and Googlebot type. Google says sites with fewer than a thousand pages should not need it. SEO teams also run their own crawlers to fetch their site the way a search engine would and find errors.

Example

Illustrative example: An online store launches 500 new product pages but links to them only from its internal search results. Googlebot cannot reach them through the navigation. Once the pages appear in an XML sitemap or are linked from category pages, Google can discover and crawl them.

FAQ

How do I get Google to crawl my site? Link new pages internally and list them in your XML sitemap. You can submit individual URLs through the URL Inspection tool in Search Console, which has a quota. Google says crawling can take from a few days to a few weeks, and requesting it does not guarantee indexing.

Can you pay Google to crawl more often? No. Google states that it does not accept payment to crawl a site more frequently or to rank it higher.

Is it legal to crawl websites? Anyone can technically fetch public pages. robots.txt expresses the site owner’s wishes, but RFC 9309 states it is not a form of access authorization. In the EU, Article 4 of Directive (EU) 2019/790 covers reproductions for text and data mining and lets rightholders reserve their content, for online content by machine-readable means.

This is not legal advice.

Last updated: October 8, 2026