Focused crawling is a crawling strategy that fetches only pages relevant to a specific topic instead of covering the whole web. The term comes from computer science research. According to Wikipedia, Chakrabarti et al. (WWW 1999) coined it, a paper the 2000 VLDB paper cites.
How does focused crawling work?
A focused crawler has to estimate how relevant a page is before it downloads it. Diligenti et al. (VLDB 2000) describe the aim as retrieving as many relevant pages as possible while visiting as few irrelevant ones as possible. Early approaches judged a link by its anchor text. Later ones use text classifiers, or context graphs that model how relevant pages are typically linked, as in the 2000 paper.
The crawler starts from seed URLs, scores the links it finds and puts the most promising ones first in its crawl queue. This saves bandwidth and storage, which is useful for niche search engines, research corpora and similar projects with limited resources.
Example
A crawler for a search engine on medical literature starts at a few specialist portals. A classifier rates each discovered link by topical fit. Only links with a high score go into the crawl queue.
Sources
Last updated: