# Diffbot

Category: [AI Bots & Crawlers](https://seo-geo-glossary.com/en/glossary/category/ai-bots-crawlers/) › [Other AI bots](https://seo-geo-glossary.com/en/glossary/category/ai-bots-crawlers/other-ai-bots/)

**Diffbot** is a company that uses machine learning to turn web pages into structured data. It runs its own crawler, which traverses the public web independently of Google and Bing and feeds a Knowledge Graph. In server logs it appears under the user agent `Diffbot`.

## What does Diffbot use its crawler for?

Diffbot describes its Knowledge Graph as a database of facts built automatically from the public web. The platform offers several interfaces, among them Extract (read entities from pages), Crawl (find entities across whole sites) and Search (query the web as a graph).

In its FAQ, Diffbot names two user agents. `Diffbot` is used for general, proactive web crawling. `Diffbot-User` is used for requests made on behalf of a person using Diffbot software. Diffbot states that it does not crawl pages for training generative AI foundation models.

## How do you control Diffbot?

By default, Diffbot's web crawls adhere to robots.txt instructions, including `Disallow` and `Crawl-delay`. The compliance can be overridden in specific cases, typically through an agreement with the site. Customers of the Crawl product can also set their own user agent.

## Example

```robots
User-agent: Diffbot
Crawl-delay: 5
Disallow: /internal/
```

## FAQ

**What is the difference between Diffbot and Diffbot-User?**
`Diffbot` crawls proactively. `Diffbot-User` fetches a URL for a person.

**Does Diffbot follow robots.txt?**
By default, yes. Diffbot says exceptions are typically limited to agreements with the site.

## At a glance

- **Operator:** Diffbot
- **Purpose:** Knowledge Graph and web search; not used to train generative foundation models, says Diffbot
- **robots.txt token:** Diffbot (user-initiated fetches: Diffbot-User)
- **Respects robots.txt:** yes by default, including Crawl-delay; may be overridden by agreement with the site
- **Docs:** [Diffbot: Does Diffbot respect robots.txt?](https://www.diffbot.com/docs/crawl/faq/robots-txt)

Last checked: October 8, 2026

## AI bots compared

| Bot | Operator | Purpose | robots.txt token | robots.txt according to the operator | IP list published? |
|---|---|---|---|---|---|
| Google-Extended | Google | Controls Gemini training and grounding (no crawler of its own) | Google-Extended | robots.txt token only; fetching is done by existing Google crawlers | n/a (no user agent of its own) |
| GPTBot | OpenAI | Training | GPTBot | Disallowing it signals content should not be used to train foundation models | yes (openai.com/gptbot.json) |
| OAI-SearchBot | OpenAI | Search (ChatGPT) | OAI-SearchBot | opted-out sites are not shown in ChatGPT search answers | yes (openai.com/searchbot.json) |
| ChatGPT-User | OpenAI | User-triggered fetch | ChatGPT-User | robots.txt rules may not apply, says OpenAI (user-initiated) | yes (openai.com/chatgpt-user.json) |
| ClaudeBot | Anthropic | Training | ClaudeBot | yes, Anthropic says its bots honor robots.txt | yes (claude.com/crawling/bots.json) |
| Claude-SearchBot | Anthropic | Search (Claude) | Claude-SearchBot | yes, Anthropic says its bots honor robots.txt | yes (claude.com/crawling/bots.json) |
| PerplexityBot | Perplexity | Search (Perplexity), not used for model training | PerplexityBot | controllable via robots.txt; the docs contain no explicit compliance statement | yes (perplexity.com/perplexitybot.json) |
| Applebot-Extended | Apple | Controls AI training (does not crawl, says Apple) | Applebot-Extended | robots.txt token only, applies to content Applebot already crawled | n/a (no crawler of its own) |
| CCBot | Common Crawl | Open web crawl archive | CCBot | can be blocked via robots.txt, says Common Crawl | yes (index.commoncrawl.org/ccbot.json) |
| Meta-ExternalAgent | Meta | Training foundation AI models, improving products | meta-externalagent | can be blocked via robots.txt, says Meta | not stated by the operator |

Sources: [OpenAI: Overview of OpenAI Crawlers](https://developers.openai.com/api/docs/bots); [Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler? (updated 2026-04-07)](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler); [Perplexity: Perplexity Crawlers](https://docs.perplexity.ai/guides/bots); [Google: Google's common crawlers (last updated 2026-07-14)](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers); [Google: Overview of Google crawlers and fetchers (last updated 2026-06-12)](https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers); [Google: Verifying Google crawlers and fetchers (IP range files)](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests); [Apple: About Applebot (2026-09-04)](https://support.apple.com/en-us/119829); [Meta: Meta Web Crawlers](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/); [Common Crawl: CCBot](https://commoncrawl.org/ccbot). Last checked: October 8, 2026

## Related terms

- [AI crawler](https://seo-geo-glossary.com/en/glossary/ai-crawler/index.md)
- [robots.txt](https://seo-geo-glossary.com/en/glossary/robots-txt/index.md)
- [Crawl-delay](https://seo-geo-glossary.com/en/glossary/crawl-delay/index.md)
- [User agent](https://seo-geo-glossary.com/en/glossary/user-agent/index.md)
- [Web scraper](https://seo-geo-glossary.com/en/glossary/web-scraper/index.md)

## Sources

- [Does Diffbot respect robots.txt? – Diffbot Docs](https://www.diffbot.com/docs/crawl/faq/robots-txt)
- [Diffbot – Wikipedia (secondary)](https://en.wikipedia.org/wiki/Diffbot)

---

Last updated: October 8, 2026
URL: https://seo-geo-glossary.com/en/glossary/diffbot/