Deutsch
Crawlers & bots

DuckDuckBot

DuckDuckBot is the web crawler of the DuckDuckGo search engine. DuckDuckGo says its job is to keep improving DuckDuckGo’s search results.

At a glance

Operator
DuckDuckGo
Purpose
crawler to improve DuckDuckGo search results
User agent
DuckDuckBot/1.1; (+http://duckduckgo.com/duckduckbot.html)
Respects robots.txt
yes, per DuckDuckGo (WWW::RobotRules)
IP list
duckduckbot.json
Docs
DuckDuckGo Help Pages

Last checked:

What role does DuckDuckBot play?

DuckDuckGo says it sources traditional links and images largely from Bing. It also runs its own crawler, DuckDuckBot, and many indexes, and adds specialized sources such as Wikipedia. AI-assisted answers use a separate bot, DuckAssistBot.

According to DuckDuckGo, DuckDuckBot respects robots.txt. DuckDuckGo lists the IP addresses it crawls from on its help page and as a JSON file, which lets you check in log files whether a request really comes from DuckDuckBot.

Example

robots.txt
User-agent: DuckDuckBot
Disallow: /internal/

User agent per DuckDuckGo: DuckDuckBot/1.1; (+http://duckduckgo.com/duckduckbot.html)

Sources

  1. DuckDuckBot – DuckDuckGo Help Pagesduckduckgo.com
  2. Where do DuckDuckGo search results come from? – DuckDuckGo Help Pagesduckduckgo.com
  3. DuckDuckBot IP list (JSON)duckduckgo.com

Last updated: