# Crawler trap Category: [Crawling & Indexing](https://seo-geo-glossary.com/en/glossary/category/crawling-indexing/) › [Crawling & crawl budget](https://seo-geo-glossary.com/en/glossary/category/crawling-indexing/crawling-crawl-budget/) · Also known as: spider trap, crawl trap, infinite space A **crawler trap** (also called a spider trap) is, in SEO, a part of a website where a crawler keeps finding new URLs that add little or no new content. Common causes are endless calendars, filter combinations, session IDs in URLs, and broken relative links. In everyday English, "spider trap" also means a trap for insects, which is what most search results for the term show. ## How do crawler traps happen? Google described the problem in 2008 as an "infinite space": a very large number of links that offer Googlebot little or nothing new to index. Its examples were a calendar with a "next month" link that can be followed forever and stores whose filters for category, price, color, and brand produce endless URL combinations. Typical causes: - **Calendars and date archives** that link indefinitely into the future or past. - **Faceted navigation and sorting**, where every combination gets its own URL. - **Session IDs or tracking parameters** in internal links, so the same content appears under ever-new addresses. - **Relative links without a leading slash**, which make paths grow longer with every hop. - **Soft 404 pages** that return a normal page with status 200 for any made-up path. - **Linked internal search results** with freely combinable search terms. ## Why do crawler traps matter for SEO? Google lists two effects of faceted URL spaces: overcrawling, because crawlers cannot tell whether a URL is useful without fetching it, and slower discovery of new, useful pages. Soft 404 pages also keep getting crawled and waste crawl budget. Google's crawl budget guide is aimed at large sites, so in our assessment a trap on a small site may cause no visible harm in Google. It does become obvious when you run your own crawler and the crawl never finishes. ## Example The page `https://www.example.com/blog/` contains this link: ```html Next ``` Under the rules for resolving relative references (RFC 3986), this becomes `https://www.example.com/blog/blog/page-2/`. If the server returns the same template there with status 200, that page contains the same link, and the next request goes to `/blog/blog/blog/page-2/`. With `href="/blog/page-2/"`, the path is absolute and the chain never starts. ## How do you find and fix crawler traps? Signs show up in a crawl of your own site (URL counts that explode in one directory, repeating path segments, long parameter chains) and in server logs where bots request large numbers of parameter URLs. Fix the cause where possible: use root-relative or absolute links, return 404 for URLs that do not exist, and limit calendars to useful date ranges. Sections that should not be crawled can be blocked in robots.txt. A `noindex` tag does little against a crawler trap, since Google still requests the page and drops it only afterward. Older guides recommend the URL Parameters tool in Search Console. Google deprecated it in 2022, stating that only about 1% of the configured parameter rules were useful for crawling. ## Common mistakes - **Empty filter combinations with status 200.** When a filter combination returns no results, Google recommends a 404 status code. Otherwise every meaningless combination becomes another crawlable page. - **Filters in changing order.** Google advises keeping the order of filters in the URL consistent and preventing duplicate filters. Otherwise each order shows the same content under a new address. - **Relying on canonical or nofollow alone.** For controlling crawl volume, Google calls `rel="canonical"` and `rel="nofollow"` generally less effective in the long term than a robots.txt block or filters in URL fragments (`#`). ## FAQ **Why does Google show insect traps for "spider trap"?** The phrase has a much more common everyday meaning. In October 2026, the top results on google.com for "spider trap" were all pest control products. "Crawler trap" is the less ambiguous term for the SEO meaning. ## At a glance - **Type:** part of a site that generates endless URLs without new content - **Typical causes:** calendars, filter combinations, session IDs, relative links - **Affects:** crawl volume and discovery of new pages - **Described by Google:** 2008, as an "infinite space" - **Docs:** [Google: Managing crawling of faceted navigation URLs](https://developers.google.com/search/docs/crawling-indexing/crawling-managing-faceted-navigation) Last checked: October 8, 2026 *Diagram: How filters multiply URLs.* Own calculation example. Three filters are enough to turn one category into thousands of URLs. Google describes two approaches, depending on whether the filtered pages are meant for search. *Diagram: Crawl budget from capacity and demand.* Google defines crawl budget as the set of URLs that Google can and wants to crawl. According to Google, the perceived URL inventory is the factor a site can control the most. ## Related terms - [Web crawler](https://seo-geo-glossary.com/en/glossary/web-crawler/index.md) - [Crawl budget](https://seo-geo-glossary.com/en/glossary/crawl-budget/index.md) - [Faceted navigation](https://seo-geo-glossary.com/en/glossary/faceted-navigation/index.md) - [URL parameters](https://seo-geo-glossary.com/en/glossary/url-parameters/index.md) - [Soft 404](https://seo-geo-glossary.com/en/glossary/soft-404/index.md) - [robots.txt](https://seo-geo-glossary.com/en/glossary/robots-txt/index.md) - [Crawler directives](https://seo-geo-glossary.com/en/glossary/crawler-directives/index.md) ## Sources - [To infinity and beyond? No! – Google Webmaster Central Blog (Aug 5, 2008)](https://developers.google.com/search/blog/2008/08/to-infinity-and-beyond-no) - [Crawl budget management – Google Crawling Infrastructure (last updated 2026-07-22)](https://developers.google.com/crawling/docs/crawl-budget) - [Managing crawling of faceted navigation URLs – Google Search Central (last updated 2025-12-18)](https://developers.google.com/search/docs/crawling-indexing/crawling-managing-faceted-navigation) - [RFC 3986, Section 5: Reference Resolution](https://www.rfc-editor.org/rfc/rfc3986#section-5) - [Spring cleaning: the URL Parameters tool – Google Search Central Blog (Mar 28, 2022)](https://developers.google.com/search/blog/2022/03/url-parameters-tool-deprecated) --- Last updated: October 8, 2026 URL: https://seo-geo-glossary.com/en/glossary/crawler-trap/