Deutsch
Types of monitoring

Website monitoring

Website monitoring is the automated, repeated testing of a website to confirm that it is available, loads quickly and works as expected. A tool runs checks on a schedule and sends an alert when a result falls outside the expected range.

How does website monitoring work?

A monitoring service requests the site from one or more locations at a fixed interval. According to reports such as Wikipedia, typical services check every one to 60 minutes. If a check fails, the service notifies the team by email, SMS, chat or URL notification (webhook). Many services repeat the check from a second location first, which reduces false alarms caused by a single network path.

Uptrends distinguishes three types of monitoring:

  • Availability monitoring checks that the site responds, including DNS and SSL certificates.
  • Performance monitoring measures load times and connection speed.
  • Functionality monitoring runs scripted steps such as a login or a checkout.
Monitoring loop: measure, compare, alertA loop in six steps. 1 Fetch: request pages or data again. 2 Compare with the previous state, for example a changed title or status code 404 instead of 200. 3 Check a rule or threshold. Without a relevant deviation there is no alert, and the next fetch follows. With a deviation, 4 an alert goes to the responsible person, 5 review whether the change is real and intended, 6 fix. Then the next fetch begins. Schematic view based on practice.no relevant deviation → no alertdeviationnext fetch1 · Fetchrequest pages ordata again2 · Comparewith the previous state,e.g. status 200 → 4043 · Ruledoes a thresholdor rule match?4 · Alertnotify theresponsible person5 · Reviewreal change or falsealarm? Intended?6 · Fixfix the cause,verify the resultSchematic view based on practice. Each team sets its own interval and rules.
Monitoring loop based on practice: fetch again, compare with the previous state, check a rule or threshold. An alert follows only for a relevant deviation, then review and fix. This is not a standard; each team sets its own interval and rules. Schematic illustration.

Where does SEO come in?

The three types above answer whether visitors can use the site. Search engines depend on further properties, for example whether robots.txt still allows crawling, whether pages carry a noindex tag or whether canonical tags point to the right URLs. A page can return a successful response and still be excluded from the index. In practice, monitoring for SEO therefore adds checks on these technical signals and on content changes (assessment; see change monitoring and SEO monitoring).

Server errors matter for search directly. Google slows down crawling when it receives 5xx or 429 responses and removes URLs from the index that persistently return a server error.

Types of monitoring compared
TypeWhat is checked?Usual frequencyWhat it catches
Uptime monitoringwhether a URL is reachable and returns the expected status code within a time limitshort intervals, often minutesOutages and server errors (5xx), including at night. On 5xx and 429, Google temporarily slows down crawling.
Website monitoring (this term)Umbrella term for availability, load time and functions of a website, such as login or checkoutvaries by checkbroken user flows on pages that respond without errors themselves
Change monitoringContent of a page or file compared with the previous fetchset by the userchanges without an error code, such as a new title, a changed canonical or a removed text section
SEO monitoringsearch-relevant signals of many URLs: status codes, redirects, noindex, canonicals, robots.txt, sitemaps, contentregular crawls, usually daily to weeklya noindex after a release that visitors never see. Google drops such pages once it reads the tag while crawling.
Rank trackingPosition of the site for set keywords, per country, language, location and devicedaily to weeklydrops for single keywords in a fixed setup. Search Console instead shows the average topmost position across all impressions.
robots.txt monitoringContent, availability and status code of robots.txtregular, interval of your choicea new blocking rule such as Disallow: /. Google generally caches robots.txt for up to 24 hours.
Sitemap monitoringAvailability, format and listed URLs of XML sitemapsregular, interval of your choicesuddenly missing URLs or whole sections. Google limits a sitemap to 50 MB (uncompressed) or 50,000 URLs.

There is no standard for these types of monitoring. Scope, frequency and examples are based on practical experience and vary by tool. Statements about Google's behaviour (crawling on 5xx and 429, noindex, robots.txt caching, sitemap limits, position in Search Console) are backed by sources. Sources: Google Search Central: Wie sich HTTP-Statuscodes und Netzwerkfehler auf die Google Suche auswirken; Google Search Central: Indexierung mit noindex blockieren / Block indexing with noindex; Google Search Central: So interpretiert Google die robots.txt-Spezifikation; Google Search Central: Sitemap erstellen und einreichen / Build and submit a sitemap; Google Search Central: So funktioniert die Google Suche / How Google Search works; Search-Console-Hilfe: Leistungsbericht (Suche). Last checked: .

Example

An online shop checks its home page, a category page and the checkout every five minutes from three regions. After a deployment the category template starts returning a 500 error. The second check confirms it, and a message reaches the on-call channel within minutes.

Common mistakes

  • Monitoring only the home page, while errors occur on templates deeper in the site.
  • Sending every alert to the same channel, so important ones get lost.
  • Treating a 200 response as proof that the page is correct.

Sources

  1. What is Website Monitoring? – Uptrendsuptrends.com
  2. Website monitoring – Wikipedia (secondary)en.wikipedia.org
  3. Pingdom uptime monitoring (vendor page)pingdom.com
  4. Fix HTTP and network errors – Google Search Centraldevelopers.google.com

Last updated: