Fresh crawl and deep crawl are historical terms from Google’s early years. The fresh crawl fetched frequently updated pages often, and the deep crawl worked through a site thoroughly by following its links. Today the two are no longer described as separate crawl types.
Where do the terms come from?
According to the Seobility wiki, Freshbot and Deepbot were the names of two crawlers. Freshbot adapted its visit frequency to how often content changed. Deepbot followed the links it could find to capture as much of a site’s structure as possible.
Matt Cutts, then at Google, wrote in 2007 that Google switched from a monthly batch update to an incremental update in summer 2003, known as “Update Fritz”. After that, parts of the index were refreshed daily or nearly daily, and Google ran a “fresh crawl” alongside its normal crawl. Cutts also recalls that when he joined Google in early 2000 there were stretches of three to four months or more without an index update, and that from mid-2000 the index was updated roughly monthly.
What applies today?
The Seobility wiki calls Freshbot and Deepbot obsolete and says SEOs rarely use the terms anymore. Google’s crawl budget documentation now describes crawl budget through two factors, the crawl capacity limit (how much Google can crawl without overloading the server) and crawl demand (how much Google wants to crawl), which includes how often pages are recrawled.
Example
In 2007, Cutts described a blog post that appeared in Google’s main search index well under an hour after it was published.
Sources
- Minty fresh indexing – Matt Cutts (August 7, 2007)mattcutts.com
- Deepbot and Freshbot – Seobility Wikiseobility.net
- Crawl budget management – Google Search Centraldevelopers.google.com
Last updated: