Deutsch
Types of monitoring

Crawl comparison

A crawl comparison puts the results of two crawls of the same website side by side. The result shows which URLs are new, which are gone and which properties changed.

What is compared?

Screaming Frog’s documentation describes three levels, and other crawlers offer similar views.

  • Changes to elements: titles, meta descriptions, crawl depth, word count, internal links and content
  • Site structure: added and missing URLs and changes at directory level
  • URL mapping: crawls with different URL patterns, for example staging against live, matched with a regular expression

Where is it used?

A single crawl is a snapshot. Only the comparison shows what changed between two states. That helps after releases and relaunches and as an ongoing check. Whether a difference is a problem still needs a human judgment, since some changes are intended.

Monitoring loop: measure, compare, alertA loop in six steps. 1 Fetch: request pages or data again. 2 Compare with the previous state, for example a changed title or status code 404 instead of 200. 3 Check a rule or threshold. Without a relevant deviation there is no alert, and the next fetch follows. With a deviation, 4 an alert goes to the responsible person, 5 review whether the change is real and intended, 6 fix. Then the next fetch begins. Schematic view based on practice.no relevant deviation → no alertdeviationnext fetch1 · Fetchrequest pages ordata again2 · Comparewith the previous state,e.g. status 200 → 4043 · Ruledoes a thresholdor rule match?4 · Alertnotify theresponsible person5 · Reviewreal change or falsealarm? Intended?6 · Fixfix the cause,verify the resultSchematic view based on practice. Each team sets its own interval and rules.
Monitoring loop based on practice: fetch again, compare with the previous state, check a rule or threshold. An alert follows only for a relevant deviation, then review and fix. This is not a standard; each team sets its own interval and rules. Schematic illustration.

Example

One crawl before and one after a template update show that the title was shortened on 3,000 product pages and the canonical tag points to a different URL on 500 pages. The example is invented.

Sources

  1. How to compare crawls - Screaming Frogscreamingfrog.co.uk

Last updated: