# CCBot

Category: [AI Bots & Crawlers](https://seo-geo-glossary.com/en/glossary/category/ai-bots-crawlers/) › [Other AI bots](https://seo-geo-glossary.com/en/glossary/category/ai-bots-crawlers/other-ai-bots/) · Also known as: Common Crawl

**CCBot** is the web crawler of Common Crawl, a non-profit that maintains an open repository of web crawl data. The archive is available at no cost to researchers, companies and individuals. The robots.txt token is `CCBot`.

## What is CCBot for?

Common Crawl describes itself as a 501(c)(3) non-profit that provides a copy of the internet for research and analysis. Because the data is freely available, other projects can use the archive without crawling the web themselves. According to reports, the name therefore comes up in discussions about AI training, although Common Crawl is not an AI provider itself.

According to Common Crawl, the user agent is `CCBot/2.0 (https://commoncrawl.org/faq/)`.

## How do you control CCBot?

CCBot obeys robots.txt. The FAQ states that it also follows the `Crawl-delay` parameter. Common Crawl publishes its IP ranges as a JSON file and uses reverse DNS entries under `crawl.commoncrawl.org`, which allows a request to be verified. Common Crawl also offers an opt-out registry.

## Example

```robots
User-agent: CCBot
Disallow: /
```

A delay instead of a block:

```robots
User-agent: CCBot
Crawl-delay: 2
```

## Common mistakes

- **Checking only the user agent.** Common Crawl notes that crawlers falsely identify themselves as CCBot. Genuine requests can be checked against the IP list and reverse DNS.

## FAQ

**Is CCBot an AI crawler?**
CCBot collects content for a general archive. Developers of AI models can use such archives, but the crawler itself does not serve one model.

**How can I tell a real CCBot?**
Check the request against the IP ranges Common Crawl publishes or the reverse DNS name. A user agent string alone can be faked.

## At a glance

- **Operator:** Common Crawl (non-profit)
- **Purpose:** open web archive
- **robots.txt token:** CCBot
- **Docs:** [Common Crawl: CCBot](https://commoncrawl.org/ccbot)
- **Respects robots.txt:** yes, including Crawl-delay
- **IP list:** [yes (JSON)](https://index.commoncrawl.org/ccbot.json)

Last checked: October 8, 2026

## AI bots compared

| Bot | Operator | Purpose | robots.txt token | robots.txt according to the operator | IP list published? |
|---|---|---|---|---|---|
| Google-Extended | Google | Controls Gemini training and grounding (no crawler of its own) | Google-Extended | robots.txt token only; fetching is done by existing Google crawlers | n/a (no user agent of its own) |
| GPTBot | OpenAI | Training | GPTBot | Disallowing it signals content should not be used to train foundation models | yes (openai.com/gptbot.json) |
| OAI-SearchBot | OpenAI | Search (ChatGPT) | OAI-SearchBot | opted-out sites are not shown in ChatGPT search answers | yes (openai.com/searchbot.json) |
| ChatGPT-User | OpenAI | User-triggered fetch | ChatGPT-User | robots.txt rules may not apply, says OpenAI (user-initiated) | yes (openai.com/chatgpt-user.json) |
| ClaudeBot | Anthropic | Training | ClaudeBot | yes, Anthropic says its bots honor robots.txt | yes (claude.com/crawling/bots.json) |
| Claude-SearchBot | Anthropic | Search (Claude) | Claude-SearchBot | yes, Anthropic says its bots honor robots.txt | yes (claude.com/crawling/bots.json) |
| PerplexityBot | Perplexity | Search (Perplexity), not used for model training | PerplexityBot | controllable via robots.txt; the docs contain no explicit compliance statement | yes (perplexity.com/perplexitybot.json) |
| Applebot-Extended | Apple | Controls AI training (does not crawl, says Apple) | Applebot-Extended | robots.txt token only, applies to content Applebot already crawled | n/a (no crawler of its own) |
| CCBot | Common Crawl | Open web crawl archive | CCBot | can be blocked via robots.txt, says Common Crawl | yes (index.commoncrawl.org/ccbot.json) |
| Meta-ExternalAgent | Meta | Training foundation AI models, improving products | meta-externalagent | can be blocked via robots.txt, says Meta | not stated by the operator |

Sources: [OpenAI: Overview of OpenAI Crawlers](https://developers.openai.com/api/docs/bots); [Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler? (updated 2026-04-07)](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler); [Perplexity: Perplexity Crawlers](https://docs.perplexity.ai/guides/bots); [Google: Google's common crawlers (last updated 2026-07-14)](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers); [Google: Overview of Google crawlers and fetchers (last updated 2026-06-12)](https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers); [Google: Verifying Google crawlers and fetchers (IP range files)](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests); [Apple: About Applebot (2026-09-04)](https://support.apple.com/en-us/119829); [Meta: Meta Web Crawlers](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/); [Common Crawl: CCBot](https://commoncrawl.org/ccbot). Last checked: October 8, 2026

## Related terms

- [Pay per crawl](https://seo-geo-glossary.com/en/glossary/pay-per-crawl/index.md)
- [Crawl-to-refer ratio](https://seo-geo-glossary.com/en/glossary/crawl-to-refer-ratio/index.md)
- [Crawl budget](https://seo-geo-glossary.com/en/glossary/crawl-budget/index.md)
- [Crawl-delay](https://seo-geo-glossary.com/en/glossary/crawl-delay/index.md)
- [robots.txt](https://seo-geo-glossary.com/en/glossary/robots-txt/index.md)
- [AI crawler](https://seo-geo-glossary.com/en/glossary/ai-crawler/index.md)

## Sources

- [CCBot – Common Crawl](https://commoncrawl.org/ccbot)
- [FAQ – Common Crawl](https://commoncrawl.org/faq)

---

Last updated: October 8, 2026
URL: https://seo-geo-glossary.com/en/glossary/ccbot/