Crawling & Indexing

Bad bots (also called rogue bots) are automated programs that access a website against the operator's wishes or ignore its rules.

Bingbot is the web crawler of Microsoft Bing.

DuckDuckBot is the web crawler of the DuckDuckGo search engine.

Googlebot is the web crawler of Google Search.

Googlebot Smartphone is the mobile version of Googlebot, Google's crawler for Search.

Google-InspectionTool is a Google crawler used by testing tools for Search, including the Rich Results Test and the URL Inspection tool in …

GoogleOther is a generic crawler from Google that product teams can use to fetch publicly accessible content.

The user agent is the software that makes a request to a web server on behalf of a user, and the name of its HTTP request header.

A web crawler (also called a crawler, spider, or bot) is a program that automatically fetches web pages and follows the links on them to …

Crawl budget is the amount of time and resources Google's crawlers spend on a site.

Crawl demand is the part of crawl budget that describes how much a crawler wants to crawl a site.

Crawl depth, also called click depth, is the number of link steps between a starting point and a URL.

Crawl errors are problems that keep a crawler from fetching a URL or a whole site, for example server errors, missing pages or failed DNS …

A crawler trap (also called a spider trap) is, in SEO, a part of a website where a crawler keeps finding new URLs that add little or no …

Crawling is the automated fetching of web pages by a program called a crawler.

Crawl rate is the number of requests a crawler sends to a website within a given period of time.

DNS (Domain Name System) translates domain names such as example.com into IP addresses so that browsers and crawlers can reach the right …

Focused crawling is a crawling strategy that fetches only pages relevant to a specific topic instead of covering the whole web.

Fresh crawl and deep crawl are historical terms from Google's early years.

An orphan page is a URL on a website that no other page of the same site links to.

A recrawl is a crawler fetching a URL it already knows again so that changes reach the index.

URL discovery is the step in which a search engine learns that a URL exists, before it fetches the page.

Crawl-delay is a robots.txt directive that asks a crawler to wait between requests.

Crawler directives are instructions that tell search engine bots how to fetch and process a website.

data-nosnippet is an HTML attribute that marks a section of text on a page as off limits for snippets in search results.

Disallow is a rule in a robots.txt file that tells a crawler not to fetch URLs whose paths begin with a given value.

max-snippet is a robots rule that sets the maximum length of a text snippet in search results, counted in characters.

The meta robots tag is an HTML element in the <head> of a page that tells search engines how to handle that page.

noarchive is a robots rule that used to stop Google from showing a cached link to a saved copy of a page in search results.

nofollow is a value that tells search engines not to follow a link or all links on a page.

noimageindex is a robots rule that tells Google not to index the images on the page where it appears.

noindex is a rule that tells search engines not to show a page or file in their search results.

nosnippet is a robots rule that tells a search engine not to show a text snippet or video preview for a page.

Paywalled content is content that is fully visible only after payment, subscription or registration.

The Robots Exclusion Protocol (REP) is the standard that lets site owners tell crawlers, through a robots.txt file, which parts of a site …

robots.txt is a plain text file at the root of a host that tells crawlers which URLs they may request.

unavailable_after is a robots rule that gives Google a date and time after which a page should no longer appear in search results.

The X-Robots-Tag is an HTTP response header that passes indexing and serving rules for a URL to search engines.

Canonicalization in SEO is the process of choosing one preferred URL when the same or very similar content is reachable under several URLs.

A canonical tag is a <link rel="canonical"> element in the <head> of a page that names the preferred URL when the same or very similar …

A cross-domain canonical is a rel="canonical" link that points to a URL on a different domain.

Duplicate content is identical or very similar content that can be reached at more than one URL, either within one website or across …

In SEO, index bloat means a website has many more URLs in a search engine's index than are useful for searchers.

Near-duplicate content is the SEO term (general usage, not a Google definition) for pages whose content is very similar to other pages …

A self-referencing canonical is a rel="canonical" link whose target is the URL of the page it sits on.

The Google Indexing API is an interface that lets site owners notify Google directly when a page is added, updated or removed.

An HTML sitemap is an ordinary page on a website that lists links to its content so visitors can find their way around.

An image sitemap is an XML sitemap that gives Google additional image URLs for a page.

IndexNow is an open protocol that lets a website tell participating search engines which URLs were added, updated or deleted, so crawlers …

lastmod is an optional field in an XML sitemap that states when the content of a URL last changed.

A news sitemap is an XML sitemap with extra details for news articles.

An RSS feed or Atom feed is an XML file that lists the latest content of a website in a machine-readable form, usually with title, link …

A sitemap index is an XML file that lists several individual sitemaps.

A video sitemap is an XML sitemap with extra details about videos embedded on pages of a site.

An XML sitemap is a file in XML format that lists the URLs of a website a search engine should crawl and index, optionally with the date …

Crawlability describes whether a crawler such as Googlebot can reach and fetch a URL.

Crawled - currently not indexed is a status in the Page indexing report of Google Search Console.

Deindexing means a URL is removed from a search engine's index and no longer shows in its results.

Discovered - currently not indexed is a status in the Page indexing report of Google Search Console.

Google cache was a Google Search feature that showed the copy of a page Google had stored.

Google Caffeine is the name of a web indexing system that Google announced in June 2010.

Indexability describes whether a URL is technically allowed to be added to a search index.

Index coverage is the indexing status of the URLs of a site in Google Search Console.

Indexed pages are URLs that Google has crawled, processed and added to its index.

Indexing, in SEO, is the step in which a search engine analyzes a crawled page and stores it in its index, the database it draws search …

Mobile-first indexing means that Google uses the mobile version of a site's content, crawled with the smartphone user agent, for indexing …

The Removals tool in Google Search Console lets site owners temporarily hide their own URLs from Google Search results.

The rendering queue is, in SEO, the waiting list where Google places crawled pages until it renders them with JavaScript.

Request indexing is a function of the URL Inspection tool in Google Search Console that asks Google to crawl a single URL.

A search index is the database in which a search engine stores what it has learned about the pages it crawled.

A soft 404 is a URL that returns a success status code (usually 200) while its content looks like an error page or an empty page.

The URL Inspection tool is a feature of Google Search Console that shows how Google has indexed a single URL and how Google can fetch and …