Perplexity-User is a Perplexity user agent that visits web pages when a person asks Perplexity a question and the answer needs a page. Each visit goes back to a single user request.
At a glance
- Operator
- Perplexity AI
- Purpose
- fetching single pages for user requests; not used for training, says Perplexity
- robots.txt token
Perplexity-User- Respects robots.txt
- generally ignores robots.txt, says Perplexity
Last checked:
How does it differ from PerplexityBot?
PerplexityBot surfaces and links websites in Perplexity search results. Perplexity recommends allowing it in robots.txt to appear there. Rules for PerplexityBot do not govern Perplexity-User, because that fetcher treats robots.txt differently.
| Bot | Operator | Purpose | robots.txt token | robots.txt according to the operator | IP list published? |
|---|---|---|---|---|---|
| Google-Extended | Controls Gemini training and grounding (no crawler of its own) | Google-Extended | robots.txt token only; fetching is done by existing Google crawlers | n/a (no user agent of its own) | |
| GPTBot | OpenAI | Training | GPTBot | Disallowing it signals content should not be used to train foundation models | yes (openai.) |
| OAI-SearchBot | OpenAI | Search (ChatGPT) | OAI-SearchBot | opted-out sites are not shown in ChatGPT search answers | yes (openai.) |
| ChatGPT-User | OpenAI | User-triggered fetch | ChatGPT-User | robots.txt rules may not apply, says OpenAI (user-initiated) | yes (openai.) |
| ClaudeBot | Anthropic | Training | ClaudeBot | yes, Anthropic says its bots honor robots.txt | yes (claude.) |
| Claude-SearchBot | Anthropic | Search (Claude) | Claude-SearchBot | yes, Anthropic says its bots honor robots.txt | yes (claude.) |
| PerplexityBot | Perplexity | Search (Perplexity), not used for model training | PerplexityBot | controllable via robots.txt; the docs contain no explicit compliance statement | yes (perplexity.) |
| Applebot-Extended | Apple | Controls AI training (does not crawl, says Apple) | Applebot-Extended | robots.txt token only, applies to content Applebot already crawled | n/a (no crawler of its own) |
| CCBot | Common Crawl | Open web crawl archive | CCBot | can be blocked via robots.txt, says Common Crawl | yes (index.) |
| Meta-ExternalAgent | Meta | Training foundation AI models, improving products | meta-externalagent | can be blocked via robots.txt, says Meta | not stated by the operator |
A selection of ten AI bots from major operators, not a complete list. All values come from each operator's own documentation (as of 8 October 2026). The table does not show whether a bot actually complies with robots.txt; it only reflects what operators state. User agents can be spoofed; published IP lists or reverse DNS identify genuine requests. Sources: OpenAI: Overview of OpenAI Crawlers; Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler? (updated 2026-04-07); Perplexity: Perplexity Crawlers; Google: Google's common crawlers (last updated 2026-07-14); Google: Overview of Google crawlers and fetchers (last updated 2026-06-12); Google: Verifying Google crawlers and fetchers (IP range files); Apple: About Applebot (2026-09-04); Meta: Meta Web Crawlers; Common Crawl: CCBot. Last checked: .
What does this mean for site owners?
Anyone who wants to restrict Perplexity-User cannot rely on robots.txt. Perplexity recommends WAF rules that combine user agent and published IP addresses, and fetching the IP lists regularly so the rules stay current. In server logs the fetcher can be recognized by the name Perplexity-User. The full user agent is Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user).
Example
User-agent: PerplexityBot
Disallow: /internal/
This rule applies to PerplexityBot only. Perplexity-User is not bound by it.
Common mistakes
- Relying on robots.txt. Perplexity writes that Perplexity-User generally ignores robots.txt rules because a user triggered the fetch.
- Assuming a PerplexityBot rule is enough. It applies to the search bot and does not cover Perplexity-User.
- Loading the IP lists only once. Perplexity recommends fetching the published IP ranges regularly and updating firewall rules automatically.
Sources
- Perplexity Crawlers – Perplexity Docsdocs.perplexity.ai
Last updated: