# Web crawler

Category: [Crawling & Indexing](https://seo-geo-glossary.com/en/glossary/category/crawling-indexing/) › [Crawlers & bots](https://seo-geo-glossary.com/en/glossary/category/crawling-indexing/crawlers-bots/) · Also known as: crawler, spider, bot, robot

A **web crawler** (also called a crawler, spider, or bot) is a program that automatically fetches web pages and follows the links on them to find more pages. Search engines use crawlers to collect content for their index. Other crawlers gather data for AI systems, web archives, or SEO audits.

## How does a web crawler work?

A crawler starts with a list of known URLs. It requests each one, processes the response, and adds newly found links to its queue. Google describes this first stage as URL discovery: some pages are known from earlier visits, others are found through links on known pages or through submitted sitemaps.

A crawler also decides how often and how fast to fetch. Google says its crawlers adjust their rate to the server's responses so they do not overload a site, and that for most sites Googlebot should not access a site more than once every few seconds on average. HTTP 500 errors, for example, tell Google to slow down. The amount of data per fetch is limited as well. By default, Google's crawlers and fetchers process the first 15 MB of a file, while Googlebot for Google Search processes the first 2 MB (64 MB for PDFs).

Before a well-behaved crawler fetches a page, it reads the site's robots.txt file. The Robots Exclusion Protocol dates back to 1994 and was published as an IETF standard, RFC 9309, in September 2022. The RFC states that these rules are not a form of access authorization. Whether a crawler follows them is up to its operator.

## Types of web crawlers

Google itself sorts its crawlers into common crawlers, special-case crawlers and user-triggered fetchers. The following grouping by purpose is a common one in SEO.

- **Search engine crawlers** such as Googlebot or Bingbot build a search index. Googlebot runs as a smartphone and a desktop crawler, and both obey the same `Googlebot` token in robots.txt.
- **AI crawlers** fetch pages for language models or AI search. OpenAI, for example, separates GPTBot (content that may be used for training foundation models) from OAI-SearchBot (results in ChatGPT search), and site owners can allow or block each one independently.
- **User-triggered fetchers** load a page because a person asked for it at that moment. OpenAI states that ChatGPT-User is not used for automatic web crawling. Google notes that its user-triggered fetchers generally ignore robots.txt.
- **SEO crawlers** are tools site owners run against their own site to find broken links, redirect chains, missing titles, and similar issues.

## Example

The following server access log line is an illustrative example with made-up values (IP, path and size). It shows which crawler requested a page:

```txt
66.249.66.1 - - [06/Oct/2026:08:14:02 +0000] "GET /products/ HTTP/1.1" 200 18342 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
```

The user-agent string proves nothing on its own, because anyone can fake it. Google recommends verifying the IP address with a reverse DNS lookup followed by a forward lookup, or by matching it against Google's published IP ranges.

## Common mistakes

- **Treating robots.txt as access control.** RFC 9309 states that its rules are not a form of access authorization, and Google says robots.txt is not a mechanism for keeping a page out of Google. Confidential content needs password protection, and `noindex` keeps a page out of the index.
- **Identifying crawlers by user agent alone.** Anyone can fake a user-agent string. Google recommends a reverse DNS lookup followed by a forward lookup, or a match against its published IP ranges.
- **Important content at the end of very large files.** When crawling for Google Search, Googlebot processes only the first 2 MB of a supported file. Anything after that is ignored.

## FAQ

**What is the difference between web crawling and web scraping?**
In common usage, crawling is about finding and fetching pages, usually at scale. Scraping is about extracting specific data from pages, such as prices or contact details. Many scrapers use a crawler to reach the pages they extract from.

**Is ChatGPT a web crawler?**
ChatGPT itself is an assistant. OpenAI runs separate crawlers and agents for it: GPTBot, OAI-SearchBot, and ChatGPT-User, each with its own robots.txt token.

**Can I block a web crawler?**
You can ask it to stay away with a robots.txt rule for its user-agent token. In practice, crawlers that ignore robots.txt have to be blocked at the server or firewall level.

## At a glance

- **Type:** program that fetches web pages automatically and follows links
- **Examples:** Googlebot, Bingbot, GPTBot, OAI-SearchBot
- **Control:** robots.txt (RFC 9309), not an access control
- **File size at Google:** first 15 MB by default; Googlebot for Search first 2 MB, PDFs 64 MB
- **Docs:** [Google: Overview of Google crawlers and fetchers](https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers)

Last checked: October 8, 2026

*Diagram: From URL to search result.* Schematic path of a URL through Google Search. Google describes three stages: crawling (including discovery and rendering), indexing and serving search results. Indexing is not guaranteed.

## Related terms

- [Crawler trap](https://seo-geo-glossary.com/en/glossary/crawler-trap/index.md)
- [Crawler directives](https://seo-geo-glossary.com/en/glossary/crawler-directives/index.md)
- [Web hosting and SEO](https://seo-geo-glossary.com/en/glossary/web-hosting-and-seo/index.md)

## Sources

- [In-depth guide to how Google Search works – Google Search Central](https://developers.google.com/search/docs/fundamentals/how-search-works)
- [Googlebot – Google Search Central (last updated 2026-02-03)](https://developers.google.com/search/docs/crawling-indexing/googlebot)
- [Overview of Google crawlers and fetchers – Google Crawling Infrastructure](https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers)
- [Verifying requests from Google crawlers and fetchers – Google Crawling Infrastructure](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests)
- [RFC 9309: Robots Exclusion Protocol (September 2022)](https://www.rfc-editor.org/rfc/rfc9309.html)
- [Overview of OpenAI crawlers – OpenAI](https://developers.openai.com/api/docs/bots)
- [Introduction to robots.txt – Google Search Central](https://developers.google.com/search/docs/crawling-indexing/robots/intro)

---

Last updated: October 8, 2026
URL: https://seo-geo-glossary.com/en/glossary/web-crawler/