# Applebot / Applebot-Extended

Category: [AI Bots & Crawlers](https://seo-geo-glossary.com/en/glossary/category/ai-bots-crawlers/) › [Other AI bots](https://seo-geo-glossary.com/en/glossary/category/ai-bots-crawlers/other-ai-bots/)

**Applebot** is Apple's web crawler. It collects content for search features in Spotlight, Siri and Safari. **Applebot-Extended** is an additional robots.txt token with which a site tells Apple whether the content Applebot has crawled may be used to train Apple's generative AI models.

## What does Apple say?

According to Apple, Applebot gathers publicly available content that powers its search features and also provides data for training the foundation models behind features such as Apple Intelligence. Applebot-Extended does not crawl pages itself. It only governs how Applebot's data may be used. Blocking it excludes training use, and the content can still appear in Apple's search features.

## Controlling the bots with robots.txt

- **Applebot** follows standard robots.txt rules. If robots.txt mentions Googlebot but not Applebot, Apple says Applebot follows the Googlebot instructions. Applebot does not follow Crawl-delay.
- **Applebot-Extended** is controlled with its own entry.

```robots
User-agent: Applebot-Extended
Disallow: /
```

## Recognizing genuine requests

User agent names can be faked, so Apple offers two checks. The reverse DNS name of a genuine request is in the `applebot.apple.com` domain. Apple also publishes its IP ranges as a JSON file.

## Further controls

Apple notes that publishers can also use `nosnippet` or `isAccessibleForFree: false` schema markup to restrict how content is used.

## Common mistakes

- **Blocking JavaScript, CSS or XHR in robots.txt.** According to Apple, pages may then not render properly.
- **Setting Crawl-delay.** According to Apple, Applebot does not follow this directive.
- **Blocking Applebot-Extended to disappear from Siri or Spotlight.** According to Apple, the token only governs use for training. Applebot-Extended does not crawl pages itself.

## FAQ

**Does blocking Applebot-Extended remove my site from Siri and Spotlight?**
According to Apple, it does not. It only affects use for training Apple's generative models.

## At a glance

- **Operator:** Apple
- **Purpose:** Search (Spotlight, Siri, Safari); data may also feed training of Apple's foundation models
- **robots.txt token:** Applebot, Applebot-Extended
- **Respects robots.txt:** yes; without its own rule Applebot follows Googlebot rules; no Crawl-delay
- **IP list:** [yes (JSON with CIDR ranges)](https://search.developer.apple.com/applebot.json)
- **Docs:** [Apple Support: About Applebot](https://support.apple.com/en-us/119829)

Last checked: October 8, 2026

## AI bots compared

| Bot | Operator | Purpose | robots.txt token | robots.txt according to the operator | IP list published? |
|---|---|---|---|---|---|
| Google-Extended | Google | Controls Gemini training and grounding (no crawler of its own) | Google-Extended | robots.txt token only; fetching is done by existing Google crawlers | n/a (no user agent of its own) |
| GPTBot | OpenAI | Training | GPTBot | Disallowing it signals content should not be used to train foundation models | yes (openai.com/gptbot.json) |
| OAI-SearchBot | OpenAI | Search (ChatGPT) | OAI-SearchBot | opted-out sites are not shown in ChatGPT search answers | yes (openai.com/searchbot.json) |
| ChatGPT-User | OpenAI | User-triggered fetch | ChatGPT-User | robots.txt rules may not apply, says OpenAI (user-initiated) | yes (openai.com/chatgpt-user.json) |
| ClaudeBot | Anthropic | Training | ClaudeBot | yes, Anthropic says its bots honor robots.txt | yes (claude.com/crawling/bots.json) |
| Claude-SearchBot | Anthropic | Search (Claude) | Claude-SearchBot | yes, Anthropic says its bots honor robots.txt | yes (claude.com/crawling/bots.json) |
| PerplexityBot | Perplexity | Search (Perplexity), not used for model training | PerplexityBot | controllable via robots.txt; the docs contain no explicit compliance statement | yes (perplexity.com/perplexitybot.json) |
| Applebot-Extended | Apple | Controls AI training (does not crawl, says Apple) | Applebot-Extended | robots.txt token only, applies to content Applebot already crawled | n/a (no crawler of its own) |
| CCBot | Common Crawl | Open web crawl archive | CCBot | can be blocked via robots.txt, says Common Crawl | yes (index.commoncrawl.org/ccbot.json) |
| Meta-ExternalAgent | Meta | Training foundation AI models, improving products | meta-externalagent | can be blocked via robots.txt, says Meta | not stated by the operator |

Sources: [OpenAI: Overview of OpenAI Crawlers](https://developers.openai.com/api/docs/bots); [Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler? (updated 2026-04-07)](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler); [Perplexity: Perplexity Crawlers](https://docs.perplexity.ai/guides/bots); [Google: Google's common crawlers (last updated 2026-07-14)](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers); [Google: Overview of Google crawlers and fetchers (last updated 2026-06-12)](https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers); [Google: Verifying Google crawlers and fetchers (IP range files)](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests); [Apple: About Applebot (2026-09-04)](https://support.apple.com/en-us/119829); [Meta: Meta Web Crawlers](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/); [Common Crawl: CCBot](https://commoncrawl.org/ccbot). Last checked: October 8, 2026

## AI bots: crawlers, tokens and controls

- **September 2022:** The IETF publishes the Robots Exclusion Protocol as RFC 9309 on the Standards Track. It defines how crawlers interpret the rules in a robots.txt file. ([Source](https://www.rfc-editor.org/rfc/rfc9309.html))
- **2023:** OpenAI introduces web crawler permissions for AI. Site owners can use them to state whether their content may be used for AI training. ([Source](https://openai.com/index/approach-to-data-and-ai/))
- **September 28, 2023:** Google introduces Google-Extended. This robots.txt token controls whether content helps improve Bard and the Vertex AI generative APIs. ([Source](https://blog.google/technology/ai/an-update-on-web-publisher-controls/))
- **February 8, 2024:** After Bard is renamed, Google's documentation lists Gemini Apps as a product affected by Google-Extended. ([Source](https://developers.google.com/search/updates#updated-the-description-of-the-google-extended-product-token_1))
- **June 10, 2024:** Apple introduces the foundation models behind Apple Intelligence. Publishers can opt out of training use, according to Apple's documentation via Applebot-Extended. ([Source](https://machinelearning.apple.com/research/introducing-apple-foundation-models))
- **July 3, 2024:** Cloudflare lets customers block AI bots, scrapers and crawlers with a single click. ([Source](https://blog.cloudflare.com/declaring-your-aindependence-block-ai-bots-scrapers-and-crawlers-with-a-single-click/))
- **July 25, 2024:** With the SearchGPT prototype, OpenAI separates search from training. Sites can appear in search results even if they opt out of AI training. ([Source](https://openai.com/index/searchgpt-prototype/))
- **August 20, 2024:** Google adds Google-CloudVertexBot to its list of crawlers. It crawls sites at the owner's request to build Vertex AI Agents. ([Source](https://developers.google.com/search/updates#introducing-the-google-cloudvertexbot-crawler))
- **April 25, 2025:** Based on publisher feedback, Google clarifies the description of the Google-Extended token. ([Source](https://developers.google.com/search/updates#updated-the-description-of-the-google-extended-product-token))
- **May 15, 2025:** Cloudflare proposes that bots and agents cryptographically sign their requests, based on HTTP Message Signatures (RFC 9421). ([Source](https://blog.cloudflare.com/web-bot-auth/))
- **July 1, 2025:** Cloudflare changes its default to block AI crawlers unless site owners decide otherwise. ([Source](https://blog.cloudflare.com/content-independence-day-no-ai-crawl-without-compensation/))
- **July 1, 2025:** Cloudflare launches pay per crawl as a private beta. Crawlers either pay per request or receive a 402 status code with a price. ([Source](https://blog.cloudflare.com/introducing-pay-per-crawl/))

## Related terms

- [Google-Extended](https://seo-geo-glossary.com/en/glossary/google-extended/index.md)

## Sources

- [About Applebot – Apple Support](https://support.apple.com/en-us/119829)
- [Applebot model training and individual privacy rights – Apple Support](https://support.apple.com/en-my/120320)

---

Last updated: October 8, 2026
URL: https://seo-geo-glossary.com/en/glossary/applebot-applebot-extended/