Amzn-SearchBot is an Amazon crawler that, according to Amazon, improves search experiences in Amazon products and services. Sites that allow it can appear in those search features; Amazon names Alexa as an example.
At a glance
- Operator
- Amazon
- Purpose
- Search in Amazon products and services (e.g. Alexa); per Amazon no generative AI training
- robots.txt token
Amzn-SearchBot- Respects robots.txt
- yes; without its own rule it follows the rules for other search bots; no Crawl-delay
- IP list
- yes, separate Amazon page
Last checked:
How does Amzn-SearchBot behave?
Amazon runs it separately from Amazonbot and Amzn-User, and each one has its own robots.txt token. If a robots.txt does not mention Amzn-SearchBot but allows other search bots, it follows the rules given to those bots. Amazon states that it does not crawl content for generative AI model training.
It fetches robots.txt per host or uses a cached copy from the last 30 days, and Amazon says changes take about 24 hours to apply. Crawl-delay is not supported. Amazon publishes a separate IP list for checking whether a request really comes from this bot.
| Bot | Operator | Purpose | robots.txt token | robots.txt according to the operator | IP list published? |
|---|---|---|---|---|---|
| Google-Extended | Controls Gemini training and grounding (no crawler of its own) | Google-Extended | robots.txt token only; fetching is done by existing Google crawlers | n/a (no user agent of its own) | |
| GPTBot | OpenAI | Training | GPTBot | Disallowing it signals content should not be used to train foundation models | yes (openai.) |
| OAI-SearchBot | OpenAI | Search (ChatGPT) | OAI-SearchBot | opted-out sites are not shown in ChatGPT search answers | yes (openai.) |
| ChatGPT-User | OpenAI | User-triggered fetch | ChatGPT-User | robots.txt rules may not apply, says OpenAI (user-initiated) | yes (openai.) |
| ClaudeBot | Anthropic | Training | ClaudeBot | yes, Anthropic says its bots honor robots.txt | yes (claude.) |
| Claude-SearchBot | Anthropic | Search (Claude) | Claude-SearchBot | yes, Anthropic says its bots honor robots.txt | yes (claude.) |
| PerplexityBot | Perplexity | Search (Perplexity), not used for model training | PerplexityBot | controllable via robots.txt; the docs contain no explicit compliance statement | yes (perplexity.) |
| Applebot-Extended | Apple | Controls AI training (does not crawl, says Apple) | Applebot-Extended | robots.txt token only, applies to content Applebot already crawled | n/a (no crawler of its own) |
| CCBot | Common Crawl | Open web crawl archive | CCBot | can be blocked via robots.txt, says Common Crawl | yes (index.) |
| Meta-ExternalAgent | Meta | Training foundation AI models, improving products | meta-externalagent | can be blocked via robots.txt, says Meta | not stated by the operator |
A selection of ten AI bots from major operators, not a complete list. All values come from each operator's own documentation (as of 8 October 2026). The table does not show whether a bot actually complies with robots.txt; it only reflects what operators state. User agents can be spoofed; published IP lists or reverse DNS identify genuine requests. Sources: OpenAI: Overview of OpenAI Crawlers; Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler? (updated 2026-04-07); Perplexity: Perplexity Crawlers; Google: Google's common crawlers (last updated 2026-07-14); Google: Overview of Google crawlers and fetchers (last updated 2026-06-12); Google: Verifying Google crawlers and fetchers (IP range files); Apple: About Applebot (2026-09-04); Meta: Meta Web Crawlers; Common Crawl: CCBot. Last checked: .
Example
User-agent: Amzn-SearchBot
Disallow: /checkout/
User agent as documented by Amazon:
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amzn-SearchBot/0.1) Chrome/W.X.Y.Z Safari/537.36
Common mistakes
- Blocking Amazonbot and forgetting Amzn-SearchBot. According to Amazon, each user agent setting is independent of the others. The search bot needs its own entry.
- Setting Crawl-delay. Amazon lists the directive as not supported.
- Expecting an immediate effect. According to Amazon, changes to robots.txt may take about 24 hours to be reflected in its systems.
FAQ
How is Amzn-SearchBot different from Amazonbot? Amazonbot crawls to improve Amazon products and services, and Amazon says its data may be used to train AI models. Amzn-SearchBot serves search features and, according to Amazon, is not used for generative AI training.
Does blocking Amazonbot also block Amzn-SearchBot?
No. Each bot has its own token. A Disallow for Amazonbot does not apply to Amzn-SearchBot unless the robots.txt names both or uses rules that cover them.
Sources
- About Amazonbot – Amazon Developerdeveloper.amazon.com
- Amzn-SearchBot IP addresses – Amazon Developerdeveloper.amazon.com
Last updated: