A staging crawl is a crawl of the staging or development version of a website, run before the new version goes live. Technical problems show up while no visitor can see them. The term is common in SEO practice and has no formal definition.
What gets checked?
Migration checklists suggest these checks on staging:
- redirects, canonical tags, XML sitemaps and structured data
- that the new URL structure matches the planned architecture
- page speed and Core Web Vitals against the values before the change
- that robots.txt blocks indexing of staging and allows it at launch (Siteimprove)
Protecting the staging site
Ryte recommends HTTP authentication so that crawlers cannot reach staging at all. Google notes that robots.txt is not a mechanism for keeping a page out of Google and names noindex and password protection instead. A crawler used for the check then needs credentials or an allowed IP address (assessment).
Example
Every page on the staging domain carries this tag:
<link rel="canonical" href="https://staging.example.com/product/">
The crawl reveals that all canonicals point to staging. Fixed before launch, the tag never reaches the live site.
More on the searchVIU blog
Sources
- Introduction to robots.txt – Google Search Centraldevelopers.google.com
- Technical SEO during migrations – Siteimprovesiteimprove.com
- 6 technical SEO mistakes to avoid in ecommerce website migration – Ryte (March 2022)en.ryte.com
Last updated: