Deutsch
Canonicals & duplicate content

Index bloat

In SEO, index bloat means a website has many more URLs in a search engine’s index than are useful for searchers. Duplicates, parameter URLs and thin pages are the usual cause. The term is industry jargon and does not appear in Google’s documentation. Outside SEO it also describes bloated database indexes, which is a different topic.

Causes and detection

Typical sources are filter and sort URLs, internal search results, date archives and tag pages. A sign of trouble is a page indexing report in Search Console that shows far more indexed URLs than the sitemap lists.

Why it matters

Google describes crawl budget as relevant mainly for sites with over a million pages, or over 10,000 pages with daily changes. On smaller sites, extra indexed URLs rarely limit crawling. Duplicate URLs can still spread signals across several addresses.

Crawl budget from crawl capacity and crawl demandTwo factors determine crawl budget. Crawl capacity is how much Google can crawl without overloading the server; it rises with stable, fast responses and drops with slow responses, 5xx errors and 429. Crawl demand is how much Google wants to crawl; it depends on the known URL inventory, popularity and staleness. Below are URL types that use up budget without benefit: duplicates, faceted URLs, soft 404s, long redirect chains and sorted variants.Crawl capacityHow much Google can crawlwithout overloading the server↑ stable, fast responses↓ slow responses, 5xx, 429Crawl demandHow much Google wants to crawl• perceived URL inventory• popularity of the URLs• staleness: picking up changesCrawl budgetthe URLs Google canand wants to crawlschematic, not a formulaUSES UP BUDGET WITHOUT BENEFITDuplicatesFacetedURLsSoft 404sLong redirectchainsSortedvariantsIf too much time goes into URLs like these, Google may not explore the rest of the site.
Schematic: crawl budget in Google's model. Index bloat itself is not a term from Google's documentation. Sources: Google Crawling: Optimize your crawl budget; Google Crawling: Managing crawling of faceted navigation URLs.

What Google advises

  • Consolidate duplicate content.
  • Fix soft 404 errors.
  • Return 404 or 410 for pages that are gone for good.
  • Use robots.txt to stop crawling of unwanted areas. Blocked URLs can still appear in the index if they are linked elsewhere.

Example (constructed)

A shop submits 20,000 product URLs in its sitemap. The page indexing report counts 180,000 indexed URLs, and the surplus consists of filter and sort combinations.

Sources

  1. Managing crawl budget for large sites – Google Search Centraldevelopers.google.com
  2. Page indexing report – Search Console Helpsupport.google.com
  3. What Is Index Bloat – Victorious (secondary source)victorious.com

Last updated: