- Crawl budget
Crawl budget is the amount of time and resources Google's crawlers spend on a site.
- Crawl demand
Crawl demand is the part of crawl budget that describes how much a crawler wants to crawl a site.
- Crawl depth
Crawl depth, also called click depth, is the number of link steps between a starting point and a URL.
- Crawl errors
Crawl errors are problems that keep a crawler from fetching a URL or a whole site, for example server errors, missing pages or failed DNS …
- Crawler trap
A crawler trap (also called a spider trap) is, in SEO, a part of a website where a crawler keeps finding new URLs that add little or no …
- Crawling
Crawling is the automated fetching of web pages by a program called a crawler.
- Crawl rate
Crawl rate is the number of requests a crawler sends to a website within a given period of time.
- DNS
DNS (Domain Name System) translates domain names such as example.com into IP addresses so that browsers and crawlers can reach the right …
- Focused crawling
Focused crawling is a crawling strategy that fetches only pages relevant to a specific topic instead of covering the whole web.
- Fresh crawl / deep crawlHistorical
Fresh crawl and deep crawl are historical terms from Google's early years.
- Orphan page
An orphan page is a URL on a website that no other page of the same site links to.
- Recrawl
A recrawl is a crawler fetching a URL it already knows again so that changes reach the index.
- URL discovery
URL discovery is the step in which a search engine learns that a URL exists, before it fetches the page.