Deutsch
Crawling & crawl budget

Crawler trap

Also known as: spider trap, crawl trap, infinite space

A crawler trap (also called a spider trap) is, in SEO, a part of a website where a crawler keeps finding new URLs that add little or no new content. Common causes are endless calendars, filter combinations, session IDs in URLs, and broken relative links. In everyday English, “spider trap” also means a trap for insects, which is what most search results for the term show.

At a glance

Type
part of a site that generates endless URLs without new content
Typical causes
calendars, filter combinations, session IDs, relative links
Affects
crawl volume and discovery of new pages
Described by Google
2008, as an "infinite space"
Docs
Google: Managing crawling of faceted navigation URLs

Last checked:

How do crawler traps happen?

Google described the problem in 2008 as an “infinite space”: a very large number of links that offer Googlebot little or nothing new to index. Its examples were a calendar with a “next month” link that can be followed forever and stores whose filters for category, price, color, and brand produce endless URL combinations.

Typical causes:

  • Calendars and date archives that link indefinitely into the future or past.
  • Faceted navigation and sorting, where every combination gets its own URL.
  • Session IDs or tracking parameters in internal links, so the same content appears under ever-new addresses.
  • Relative links without a leading slash, which make paths grow longer with every hop.
  • Soft 404 pages that return a normal page with status 200 for any made-up path.
  • Linked internal search results with freely combinable search terms.
How three filters multiply one category into thousands of URLsOwn calculation example: the category /shoes has three filters with 8 colors, 12 sizes and 20 brands. With all three filters set there are 8 × 12 × 20 = 1,920 URLs. If filters may stay empty, it is 9 × 13 × 21 = 2,457, and with three sort orders 7,371. Google describes two paths: URLs meant for search keep a fixed filter order, no duplicate filters and a 404 for empty results; URLs not meant for search are blocked in robots.txt or keep filters in the URL fragment.Category1 page/shoesColor8 values?color=redSize12 values&size=9Brand20 values&brand=…1 URLcategory8 URLswith color96 URLs+ size1,920 URLsall three filtersOWN CALCULATION EXAMPLE8 × 12 × 20 = 1,920 URLs with all three filters setFilters may stay empty: 9 × 13 × 21 = 2,457 URLsPlus 3 sort orders (?sort=price …): 2,457 × 3 = 7,371 URLsEvery other parameter order adds yet another URL.Meant for searchfixed filter order,no duplicate filters,404 for empty resultsNot meant for searchdisallow in robots.txtor keep filters in theURL fragment (#color=red)
Own calculation example. Three filters are enough to turn one category into thousands of URLs. Google describes two approaches, depending on whether the filtered pages are meant for search. Sources: Google Crawling: Managing crawling of faceted navigation URLs; Google Crawling: Optimize your crawl budget.

Why do crawler traps matter for SEO?

Google lists two effects of faceted URL spaces: overcrawling, because crawlers cannot tell whether a URL is useful without fetching it, and slower discovery of new, useful pages. Soft 404 pages also keep getting crawled and waste crawl budget. Google’s crawl budget guide is aimed at large sites, so in our assessment a trap on a small site may cause no visible harm in Google. It does become obvious when you run your own crawler and the crawl never finishes.

Crawl budget from crawl capacity and crawl demandTwo factors determine crawl budget. Crawl capacity is how much Google can crawl without overloading the server; it rises with stable, fast responses and drops with slow responses, 5xx errors and 429. Crawl demand is how much Google wants to crawl; it depends on the known URL inventory, popularity and staleness. Below are URL types that use up budget without benefit: duplicates, faceted URLs, soft 404s, long redirect chains and sorted variants.Crawl capacityHow much Google can crawlwithout overloading the server↑ stable, fast responses↓ slow responses, 5xx, 429Crawl demandHow much Google wants to crawl• perceived URL inventory• popularity of the URLs• staleness: picking up changesCrawl budgetthe URLs Google canand wants to crawlschematic, not a formulaUSES UP BUDGET WITHOUT BENEFITDuplicatesFacetedURLsSoft 404sLong redirectchainsSortedvariantsIf too much time goes into URLs like these, Google may not explore the rest of the site.
Google defines crawl budget as the set of URLs that Google can and wants to crawl. According to Google, the perceived URL inventory is the factor a site can control the most. Sources: Google Crawling: Optimize your crawl budget; Google Crawling: Managing crawling of faceted navigation URLs.

Example

The page https://www.example.com/blog/ contains this link:

HTML
<a href="blog/page-2/">Next</a>

Under the rules for resolving relative references (RFC 3986), this becomes https://www.example.com/blog/blog/page-2/. If the server returns the same template there with status 200, that page contains the same link, and the next request goes to /blog/blog/blog/page-2/. With href="/blog/page-2/", the path is absolute and the chain never starts.

How do you find and fix crawler traps?

Signs show up in a crawl of your own site (URL counts that explode in one directory, repeating path segments, long parameter chains) and in server logs where bots request large numbers of parameter URLs.

Fix the cause where possible: use root-relative or absolute links, return 404 for URLs that do not exist, and limit calendars to useful date ranges. Sections that should not be crawled can be blocked in robots.txt. A noindex tag does little against a crawler trap, since Google still requests the page and drops it only afterward.

Older guides recommend the URL Parameters tool in Search Console. Google deprecated it in 2022, stating that only about 1% of the configured parameter rules were useful for crawling.

Common mistakes

  • Empty filter combinations with status 200. When a filter combination returns no results, Google recommends a 404 status code. Otherwise every meaningless combination becomes another crawlable page.
  • Filters in changing order. Google advises keeping the order of filters in the URL consistent and preventing duplicate filters. Otherwise each order shows the same content under a new address.
  • Relying on canonical or nofollow alone. For controlling crawl volume, Google calls rel="canonical" and rel="nofollow" generally less effective in the long term than a robots.txt block or filters in URL fragments (#).

FAQ

Why does Google show insect traps for “spider trap”? The phrase has a much more common everyday meaning. In October 2026, the top results on google.com for “spider trap” were all pest control products. “Crawler trap” is the less ambiguous term for the SEO meaning.

Sources

  1. To infinity and beyond? No! – Google Webmaster Central Blog (Aug 5, 2008)developers.google.com
  2. Crawl budget management – Google Crawling Infrastructure (last updated 2026-07-22)developers.google.com
  3. Managing crawling of faceted navigation URLs – Google Search Central (last updated 2025-12-18)developers.google.com
  4. RFC 3986, Section 5: Reference Resolutionrfc-editor.org
  5. Spring cleaning: the URL Parameters tool – Google Search Central Blog (Mar 28, 2022)developers.google.com

Last updated: