A search index is the database in which a search engine stores what it has learned about the pages it crawled. The Google index is the one Google draws on for every query. A page that is not in the index cannot show up in Google Search results.
At a glance
- Type
- database a search engine draws its results from
- Size (according to Google)
- hundreds of billions of webpages, well over 100,000,000 GB
- Basis at Google
- mobile version of a page (mobile-first indexing)
- Check with
- URL Inspection tool and Page indexing report in Search Console
Last checked:
How does a page get into the index?
Google describes three stages. During crawling, Googlebot downloads text, images and videos from pages it found through links, known URLs or sitemaps. During indexing, Google analyzes that content and stores it in the index, which Google calls “a large database hosted on thousands of computers.” The third stage, serving, returns matching pages from the index when someone searches.
While indexing, Google groups pages with similar content into clusters and picks the most representative one as the canonical URL. That URL is the one that usually appears in results. Google’s own description compares the index to the index at the back of a book and puts its size at hundreds of billions of webpages and well over 100,000,000 gigabytes.
Crawling is not strictly required for a URL to be indexed. The Search Console glossary notes that Google can index a URL without reading its content, for example when robots.txt blocks it but other pages link to it.
Is every page indexed?
No. Google states plainly that “indexing isn’t guaranteed.” Common reasons a URL stays out are a noindex rule, a different canonical URL chosen for a duplicate, low content quality or technical errors such as non-200 status codes.
Missing URLs are not automatically a problem. The Search Console Help says the goal is to get the canonical version of every important page indexed and warns site owners not to expect every URL to be indexed.
Example
An online store has 50,000 URLs. 30,000 of them are filter combinations whose canonical tag points to the category page. In the Search Console page indexing report, those 30,000 appear as “Alternate page with proper canonical tag.” That is intended. Product pages listed as “Crawled - currently not indexed” would deserve a closer look.
Common mistakes
- Expecting every URL to be indexed. Search Console Help explicitly says not to. Duplicates or pages without meaningful content do not need to be indexed, the key pages do.
- Using
site:as a count. An indexed URL can show up for asite:query, but Google says this is not guaranteed. The URL Inspection tool and the Page indexing report are more reliable. - Treating the Removals tool as permanent. A request lasts only about six months. After that, the URL can reappear unless it was also removed with
noindex, a 404/410, or password protection.
FAQ
How do I check whether a page is in Google’s index?
The URL Inspection tool in Search Console shows the status of a single URL. The page indexing report covers the whole property and lists a reason for every URL that is not indexed. A site: search gives a rough impression but no complete count.
How do I remove a page from the index?
Permanently, by adding noindex, by returning a 404 or 410 status code, or by putting the page behind a login. The Removals tool in Search Console hides a URL for only about six months.
Sources
- In-depth guide to how Google Search works – Google Search Centraldevelopers.google.com
- How Google Search organizes information – Google Searchgoogle.com
- Indexing – Search Console Help (glossary)support.google.com
- Page indexing report – Search Console Helpsupport.google.com
- Removals and SafeSearch reports tool – Search Console Helpsupport.google.com
- site: search operator – Google Search Centraldevelopers.google.com
Last updated: