# PerplexityBot

Category: [AI Bots & Crawlers](https://seo-geo-glossary.com/en/glossary/category/ai-bots-crawlers/) › [OpenAI, Anthropic & Perplexity bots](https://seo-geo-glossary.com/en/glossary/category/ai-bots-crawlers/openai-anthropic-perplexity/)

**PerplexityBot** is the crawler of the AI search engine Perplexity. According to Perplexity, it is designed to surface and link websites in Perplexity's search results. Perplexity states that it does not use PerplexityBot to crawl content for AI foundation models.

## How does PerplexityBot identify itself?

Perplexity documents this user agent:

```txt
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)
```

Perplexity publishes the bot's IP ranges as a JSON file at `perplexity.com/perplexitybot.json` and recommends combining user-agent matching with IP verification, for example in firewall rules. Since a user agent can be spoofed, the IP check is the more reliable way to recognize genuine requests in log files.

Perplexity runs a second agent, **Perplexity-User**. It fetches pages when a person asks Perplexity a question and, according to Perplexity, generally ignores robots.txt because the request comes from a user.

## Controlling PerplexityBot with robots.txt

To block PerplexityBot from a whole site:

```robots
User-agent: PerplexityBot
Disallow: /
```

Perplexity says changes can take up to 24 hours to be reflected in its systems. To appear in Perplexity's search results, Perplexity recommends allowing PerplexityBot in robots.txt and permitting requests from its published IP ranges. Fetches by Perplexity-User are not covered by a PerplexityBot rule.

## Common mistakes

- **Blocking PerplexityBot to stop all Perplexity traffic.** The rule covers only the crawler. User-triggered fetches by Perplexity-User continue.
- **Trusting the user agent alone.** Requests that only claim to be PerplexityBot distort log reports. The published IP list shows which ones are real.
- **Forgetting firewall rules.** Perplexity recommends allowing its published IP ranges and keeping the list updated automatically. A WAF rule that blocks them effectively overrides an allow in robots.txt (our inference, not a Perplexity statement).

## Criticism

In August 2025, Cloudflare accused Perplexity of using undeclared crawlers with changing user agents and IP addresses to get around robots.txt blocks, in addition to its declared bots. Cloudflare removed Perplexity from its list of verified bots. Perplexity rejected the accusations and argued that fetches triggered by users come from AI assistants and should not be treated as crawlers.

## Frequently asked questions

**What is the difference between PerplexityBot and Perplexity-User?**
PerplexityBot crawls pages in advance for Perplexity's search results and can be controlled through robots.txt, although the docs do not explicitly promise compliance. Perplexity-User loads a page at the moment a person asks about it and, according to Perplexity, generally does not follow robots.txt.

**How do I find PerplexityBot in my logs?**
Look for `PerplexityBot/1.0` in the user agent. Whether the request is genuine shows only when the IP address matches the published JSON list.

## At a glance

- **Operator:** Perplexity
- **Purpose:** surface websites in Perplexity search results, not used for training (per Perplexity)
- **robots.txt token:** PerplexityBot
- **Respects robots.txt:** controllable via robots.txt; the docs make no explicit compliance statement
- **IP list:** [perplexity.com/perplexitybot.json](https://www.perplexity.com/perplexitybot.json)
- **Docs:** [Perplexity: Perplexity crawlers](https://docs.perplexity.ai/docs/resources/perplexity-crawlers)

Last checked: October 8, 2026

## AI bots compared

| Bot | Operator | Purpose | robots.txt token | robots.txt according to the operator | IP list published? |
|---|---|---|---|---|---|
| Google-Extended | Google | Controls Gemini training and grounding (no crawler of its own) | Google-Extended | robots.txt token only; fetching is done by existing Google crawlers | n/a (no user agent of its own) |
| GPTBot | OpenAI | Training | GPTBot | Disallowing it signals content should not be used to train foundation models | yes (openai.com/gptbot.json) |
| OAI-SearchBot | OpenAI | Search (ChatGPT) | OAI-SearchBot | opted-out sites are not shown in ChatGPT search answers | yes (openai.com/searchbot.json) |
| ChatGPT-User | OpenAI | User-triggered fetch | ChatGPT-User | robots.txt rules may not apply, says OpenAI (user-initiated) | yes (openai.com/chatgpt-user.json) |
| ClaudeBot | Anthropic | Training | ClaudeBot | yes, Anthropic says its bots honor robots.txt | yes (claude.com/crawling/bots.json) |
| Claude-SearchBot | Anthropic | Search (Claude) | Claude-SearchBot | yes, Anthropic says its bots honor robots.txt | yes (claude.com/crawling/bots.json) |
| PerplexityBot | Perplexity | Search (Perplexity), not used for model training | PerplexityBot | controllable via robots.txt; the docs contain no explicit compliance statement | yes (perplexity.com/perplexitybot.json) |
| Applebot-Extended | Apple | Controls AI training (does not crawl, says Apple) | Applebot-Extended | robots.txt token only, applies to content Applebot already crawled | n/a (no crawler of its own) |
| CCBot | Common Crawl | Open web crawl archive | CCBot | can be blocked via robots.txt, says Common Crawl | yes (index.commoncrawl.org/ccbot.json) |
| Meta-ExternalAgent | Meta | Training foundation AI models, improving products | meta-externalagent | can be blocked via robots.txt, says Meta | not stated by the operator |

Sources: [OpenAI: Overview of OpenAI Crawlers](https://developers.openai.com/api/docs/bots); [Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler? (updated 2026-04-07)](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler); [Perplexity: Perplexity Crawlers](https://docs.perplexity.ai/guides/bots); [Google: Google's common crawlers (last updated 2026-07-14)](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers); [Google: Overview of Google crawlers and fetchers (last updated 2026-06-12)](https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers); [Google: Verifying Google crawlers and fetchers (IP range files)](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests); [Apple: About Applebot (2026-09-04)](https://support.apple.com/en-us/119829); [Meta: Meta Web Crawlers](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/); [Common Crawl: CCBot](https://commoncrawl.org/ccbot). Last checked: October 8, 2026

## AI bots: crawlers, tokens and controls

- **September 2022:** The IETF publishes the Robots Exclusion Protocol as RFC 9309 on the Standards Track. It defines how crawlers interpret the rules in a robots.txt file. ([Source](https://www.rfc-editor.org/rfc/rfc9309.html))
- **2023:** OpenAI introduces web crawler permissions for AI. Site owners can use them to state whether their content may be used for AI training. ([Source](https://openai.com/index/approach-to-data-and-ai/))
- **September 28, 2023:** Google introduces Google-Extended. This robots.txt token controls whether content helps improve Bard and the Vertex AI generative APIs. ([Source](https://blog.google/technology/ai/an-update-on-web-publisher-controls/))
- **February 8, 2024:** After Bard is renamed, Google's documentation lists Gemini Apps as a product affected by Google-Extended. ([Source](https://developers.google.com/search/updates#updated-the-description-of-the-google-extended-product-token_1))
- **June 10, 2024:** Apple introduces the foundation models behind Apple Intelligence. Publishers can opt out of training use, according to Apple's documentation via Applebot-Extended. ([Source](https://machinelearning.apple.com/research/introducing-apple-foundation-models))
- **July 3, 2024:** Cloudflare lets customers block AI bots, scrapers and crawlers with a single click. ([Source](https://blog.cloudflare.com/declaring-your-aindependence-block-ai-bots-scrapers-and-crawlers-with-a-single-click/))
- **July 25, 2024:** With the SearchGPT prototype, OpenAI separates search from training. Sites can appear in search results even if they opt out of AI training. ([Source](https://openai.com/index/searchgpt-prototype/))
- **August 20, 2024:** Google adds Google-CloudVertexBot to its list of crawlers. It crawls sites at the owner's request to build Vertex AI Agents. ([Source](https://developers.google.com/search/updates#introducing-the-google-cloudvertexbot-crawler))
- **April 25, 2025:** Based on publisher feedback, Google clarifies the description of the Google-Extended token. ([Source](https://developers.google.com/search/updates#updated-the-description-of-the-google-extended-product-token))
- **May 15, 2025:** Cloudflare proposes that bots and agents cryptographically sign their requests, based on HTTP Message Signatures (RFC 9421). ([Source](https://blog.cloudflare.com/web-bot-auth/))
- **July 1, 2025:** Cloudflare changes its default to block AI crawlers unless site owners decide otherwise. ([Source](https://blog.cloudflare.com/content-independence-day-no-ai-crawl-without-compensation/))
- **July 1, 2025:** Cloudflare launches pay per crawl as a private beta. Crawlers either pay per request or receive a 402 status code with a price. ([Source](https://blog.cloudflare.com/introducing-pay-per-crawl/))

## Related terms

- [Perplexity-User](https://seo-geo-glossary.com/en/glossary/perplexity-user/index.md)
- [AI crawler](https://seo-geo-glossary.com/en/glossary/ai-crawler/index.md)
- [AI search crawler](https://seo-geo-glossary.com/en/glossary/ai-search-crawler/index.md)
- [robots.txt](https://seo-geo-glossary.com/en/glossary/robots-txt/index.md)
- [Crawler IP ranges](https://seo-geo-glossary.com/en/glossary/crawler-ip-ranges/index.md)
- [Bot verification](https://seo-geo-glossary.com/en/glossary/bot-verification/index.md)

## Sources

- [Perplexity crawlers – Perplexity Docs](https://docs.perplexity.ai/docs/resources/perplexity-crawlers)
- [PerplexityBot IP ranges (JSON) – Perplexity](https://www.perplexity.com/perplexitybot.json)
- [Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives – Cloudflare Blog, 2025-08-04](https://blog.cloudflare.com/perplexity-is-using-stealth-undeclared-crawlers-to-evade-website-no-crawl-directives/)
- [Cloudflare Accuses AI Startup of 'Stealth Crawling Behavior' – TechRepublic (secondary source for Perplexity's response)](https://www.techrepublic.com/article/news-cloudflare-accuses-perplexity-stealth-crawling-violations/)
- [Agents or Bots? Making Sense of AI on the Open Web – Perplexity (2025-08-04)](https://www.perplexity.ai/hub/blog/agents-or-bots-making-sense-of-ai-on-the-open-web)

---

Last updated: October 8, 2026
URL: https://seo-geo-glossary.com/en/glossary/perplexitybot/