DuckAssistBot is the web crawler DuckDuckGo uses for the AI-assisted answers in DuckDuckGo Search. According to DuckDuckGo, it crawls pages in real time so that those answers can cite their sources.
At a glance
- Operator
- DuckDuckGo
- Purpose
- Fetches for AI-assisted answers, not used for training
- robots.txt token
- DuckAssistBot
- User agent
- DuckAssistBot/1.2; (+http://duckduckgo.com/duckassistbot.html)
- Respects robots.txt
- can be blocked via robots.txt, takes effect after 72 hours
- IP list
- yes, as a JSON file
Last checked:
What does DuckAssistBot do?
DuckDuckGo states that the data is not used to train AI models. The bot identifies itself with the user agent DuckAssistBot/1.2; (+http://duckduckgo.com/duckassistbot.html). DuckDuckGo publishes the IP addresses it crawls from, 486 entries as of October 2026, as a JSON file.
DuckDuckGo’s classic search crawler is DuckDuckBot, a separate bot with a different job.
| Bot | Operator | Purpose | robots.txt token | robots.txt according to the operator | IP list published? |
|---|---|---|---|---|---|
| Google-Extended | Controls Gemini training and grounding (no crawler of its own) | Google-Extended | robots.txt token only; fetching is done by existing Google crawlers | n/a (no user agent of its own) | |
| GPTBot | OpenAI | Training | GPTBot | Disallowing it signals content should not be used to train foundation models | yes (openai.) |
| OAI-SearchBot | OpenAI | Search (ChatGPT) | OAI-SearchBot | opted-out sites are not shown in ChatGPT search answers | yes (openai.) |
| ChatGPT-User | OpenAI | User-triggered fetch | ChatGPT-User | robots.txt rules may not apply, says OpenAI (user-initiated) | yes (openai.) |
| ClaudeBot | Anthropic | Training | ClaudeBot | yes, Anthropic says its bots honor robots.txt | yes (claude.) |
| Claude-SearchBot | Anthropic | Search (Claude) | Claude-SearchBot | yes, Anthropic says its bots honor robots.txt | yes (claude.) |
| PerplexityBot | Perplexity | Search (Perplexity), not used for model training | PerplexityBot | controllable via robots.txt; the docs contain no explicit compliance statement | yes (perplexity.) |
| Applebot-Extended | Apple | Controls AI training (does not crawl, says Apple) | Applebot-Extended | robots.txt token only, applies to content Applebot already crawled | n/a (no crawler of its own) |
| CCBot | Common Crawl | Open web crawl archive | CCBot | can be blocked via robots.txt, says Common Crawl | yes (index.) |
| Meta-ExternalAgent | Meta | Training foundation AI models, improving products | meta-externalagent | can be blocked via robots.txt, says Meta | not stated by the operator |
A selection of ten AI bots from major operators, not a complete list. All values come from each operator's own documentation (as of 8 October 2026). The table does not show whether a bot actually complies with robots.txt; it only reflects what operators state. User agents can be spoofed; published IP lists or reverse DNS identify genuine requests. Sources: OpenAI: Overview of OpenAI Crawlers; Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler? (updated 2026-04-07); Perplexity: Perplexity Crawlers; Google: Google's common crawlers (last updated 2026-07-14); Google: Overview of Google crawlers and fetchers (last updated 2026-06-12); Google: Verifying Google crawlers and fetchers (IP range files); Apple: About Applebot (2026-09-04); Meta: Meta Web Crawlers; Common Crawl: CCBot. Last checked: .
How can you control DuckAssistBot?
A site can disallow the user agent in robots.txt. DuckDuckGo says a change takes effect after 72 hours and that opting out does not affect organic rankings or whether the site appears in DuckDuckGo search results. Some third-party bot directories describe blocking it as only an opt-out from AI training, which does not match DuckDuckGo’s description.
Example
User-agent: DuckAssistBot
Disallow: /internal/
The rule blocks only the /internal/ directory for this bot. All other paths stay open.
FAQ
Does DuckAssistBot train AI models? DuckDuckGo says no. Its help page states that the data is not used in any way to train AI models.
Sources
- DuckAssistBot – DuckDuckGo Help Pagesduckduckgo.com
Last updated: