Deutsch
Canonicals & duplicate content

Duplicate content

Also known as: duplicate pages, duplicated content

Duplicate content is identical or very similar content that can be reached at more than one URL, either within one website or across several domains. Google groups such URLs, picks one as the canonical URL and usually shows only that one in search results.

At a glance

Type
identical or very similar content at more than one URL
Affects
choice of the canonical URL, crawling
Penalty?
none for technical duplicates (according to Google)
Canonicalization signals
301 and rel="canonical" strong, sitemap weak
Docs
Google: What is URL canonicalization

Last checked:

How does Google handle duplicate content?

Google clusters pages with the same main content and selects the one it considers most complete and useful as canonical. Google calls this process canonicalization. The other URLs in the cluster count as duplicates and, according to Google, are crawled less often to reduce load on the site.

There is no penalty attached. Google writes that some duplicate content on a site is normal and not a violation of its spam policies. The practical issues are different. Google may pick a URL you did not intend, it gets harder to track how your content performs in search results, and on large sites crawling duplicates takes time away from pages that matter.

The exception is copying other sites’ content to manipulate rankings. Google’s spam policies treat republishing scraped content without adding value as spam.

Duplicate cluster and canonical URLFour URL variants of the same category page form a cluster: an HTTP version, a version without www, one with the tracking parameter gclid and one with a sort parameter. Google picks https://www.example.com/dresses/ as the canonical URL. Below are the signals in order of strength according to Google: redirect strong, rel="canonical" strong, sitemap weak.DUPLICATE CLUSTERCANONICAL URLhttp://www.example.com/dresses/protocol variant (HTTP)https://example.com/dresses/host variant (no www)https://www.example.com/dresses/?gclid=ABCDtracking parameterhttps://www.example.com/dresses/?sort=pricesort function (parameter)Google pickshttps://www.example.com/dresses/usually shown in resultsand crawled mostregularlyWHAT INFLUENCES THE CHOICE, BY STRENGTH ACCORDING TO GOOGLE1redirect to the preferred URLstrong2rel="canonical" (HTML link or HTTP header)strong3URL listed in the sitemapweakGoogle also prefers HTTPS over HTTP.Schematic. The preference is a hint. Google may choose a different URL as canonical.Several signals pointing to the same URL are stronger together.
Google groups identical or very similar pages into a cluster and picks one canonical URL. According to Google, redirects and rel="canonical" are strong signals and the sitemap is a weak one. Your own preference remains a hint. Sources: Google Search Central: What is URL canonicalization; Google Search Central: How to specify a canonical URL with rel="canonical" and other methods.

What causes duplicate content?

Most duplicates are a side effect of how a site is built. Google names several typical sources:

  • protocol and host variants (HTTP and HTTPS, with and without www)
  • URL parameters for filters, sorting or tracking
  • regional variants with the same text, such as US and UK pages
  • separate mobile and desktop URLs
  • demo or test environments that are accidentally crawlable

CMS archives, tag pages, print versions and manufacturer descriptions reused by many online stores add to the list.

Example

The same product page is reachable at three URLs:

Text
https://www.example.com/shoe-123
https://www.example.com/shoe-123?color=blue
http://example.com/shoe-123

The HTTP and non-www versions redirect with a 301 to the first URL. The parameter URL stays reachable and carries a canonical tag:

HTML
<link rel="canonical" href="https://www.example.com/shoe-123">

How do you fix duplicate content?

Google lists signals of different strength. Redirects and rel="canonical" are strong signals, sitemap inclusion is a weak one. All of them are hints. The most reliable setup is one where redirects, canonical tags, sitemaps and internal links all point to the same URL.

Common mistakes

  • Blocking duplicates in robots.txt. Google advises against using robots.txt for canonicalization. Google may still index disallowed URLs, just without their content.
  • Using the URL removal tool. The removal tool hides all versions of a URL from Search. Google says it is not meant for canonicalization.
  • Conflicting signals. If the sitemap lists one URL and rel="canonical" on the same page names another, the hints contradict each other. Google recommends pointing every method to the same URL.
  • Declaring a URL fragment canonical. Google generally does not support fragments such as #color-blue, so a URL with a fragment does not work as a canonical URL.

FAQ

Does Google penalize duplicate content? Not for technical duplicates within a site. Content copied from other sites to manipulate rankings falls under Google’s spam policies.

How do I find duplicate content on my site? The Search Console page indexing report lists URLs with statuses such as “Duplicate without user-selected canonical.” A crawl of the site reveals identical titles, identical body text and parameter variants.

Sources

  1. What is URL canonicalization – Google Search Centraldevelopers.google.com
  2. How to specify a canonical URL with rel="canonical" and other methods – Google Search Centraldevelopers.google.com
  3. Spam policies for Google web search (scraping) – Google Search Centraldevelopers.google.com
  4. Page indexing report – Search Console Helpsupport.google.com

Last updated: