Amazonbot is Amazon’s web crawler. According to Amazon it is used to improve Amazon products and services, and the data may be used to train Amazon AI models. The robots.txt token is Amazonbot.
At a glance
- Operator
- Amazon
- Purpose
- Improving Amazon products and services; per Amazon also AI model training
- robots.txt token
Amazonbot- Respects robots.txt
- yes, Robots Exclusion Protocol; no Crawl-delay
- IP list
- yes, separate Amazon page
Last checked:
What is Amazonbot for?
Amazon describes two more bots on the same page. Amzn-SearchBot improves search in Amazon products and services and makes content eligible for features such as Alexa. Amzn-User supports user actions, for example when Alexa needs current information to answer a question. Amazon says that Amzn-SearchBot does not crawl for generative AI training.
| Bot | Operator | Purpose | robots.txt token | robots.txt according to the operator | IP list published? |
|---|---|---|---|---|---|
| Google-Extended | Controls Gemini training and grounding (no crawler of its own) | Google-Extended | robots.txt token only; fetching is done by existing Google crawlers | n/a (no user agent of its own) | |
| GPTBot | OpenAI | Training | GPTBot | Disallowing it signals content should not be used to train foundation models | yes (openai.) |
| OAI-SearchBot | OpenAI | Search (ChatGPT) | OAI-SearchBot | opted-out sites are not shown in ChatGPT search answers | yes (openai.) |
| ChatGPT-User | OpenAI | User-triggered fetch | ChatGPT-User | robots.txt rules may not apply, says OpenAI (user-initiated) | yes (openai.) |
| ClaudeBot | Anthropic | Training | ClaudeBot | yes, Anthropic says its bots honor robots.txt | yes (claude.) |
| Claude-SearchBot | Anthropic | Search (Claude) | Claude-SearchBot | yes, Anthropic says its bots honor robots.txt | yes (claude.) |
| PerplexityBot | Perplexity | Search (Perplexity), not used for model training | PerplexityBot | controllable via robots.txt; the docs contain no explicit compliance statement | yes (perplexity.) |
| Applebot-Extended | Apple | Controls AI training (does not crawl, says Apple) | Applebot-Extended | robots.txt token only, applies to content Applebot already crawled | n/a (no crawler of its own) |
| CCBot | Common Crawl | Open web crawl archive | CCBot | can be blocked via robots.txt, says Common Crawl | yes (index.) |
| Meta-ExternalAgent | Meta | Training foundation AI models, improving products | meta-externalagent | can be blocked via robots.txt, says Meta | not stated by the operator |
A selection of ten AI bots from major operators, not a complete list. All values come from each operator's own documentation (as of 8 October 2026). The table does not show whether a bot actually complies with robots.txt; it only reflects what operators state. User agents can be spoofed; published IP lists or reverse DNS identify genuine requests. Sources: OpenAI: Overview of OpenAI Crawlers; Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler? (updated 2026-04-07); Perplexity: Perplexity Crawlers; Google: Google's common crawlers (last updated 2026-07-14); Google: Overview of Google crawlers and fetchers (last updated 2026-06-12); Google: Verifying Google crawlers and fetchers (IP range files); Apple: About Applebot (2026-09-04); Meta: Meta Web Crawlers; Common Crawl: CCBot. Last checked: .
How do you control Amazonbot?
Amazon states that Amazonbot respects the Robots Exclusion Protocol and honors user agent directives. It also respects the page-level noarchive robots meta tag, which Amazon says keeps a page out of model training. If a fresh robots.txt cannot be fetched, a cached copy from the last 30 days is used.
Crawl-delay is not supported by Amazonbot, Amzn-SearchBot or Amzn-User. For Amzn-User, Amazon notes that it may not follow all robots.txt directives, because the requests are user-initiated.
Example
User-agent: Amazonbot
Disallow: /internal/
And as a robots meta tag on a single page:
<meta name="robots" content="noarchive">
Common mistakes
- Setting Crawl-delay. Amazon lists the directive as not supported for Amazonbot, Amzn-SearchBot and Amzn-User.
- Blocking only Amazonbot. According to Amazon, each user agent setting is independent of the others. Amzn-SearchBot and Amzn-User need their own entries.
- Expecting an immediate effect. According to Amazon, changes to robots.txt may take about 24 hours to be reflected in its systems.
FAQ
Can I limit Amazonbot’s request rate with Crawl-delay? No. Amazon lists the directive as not supported.
Does robots.txt control every Amazon bot? Not fully. Amazon says Amzn-User may not follow all directives.
Sources
- Amazonbot – Amazon Developerdeveloper.amazon.com
Last updated: