- Bad bots
Bad bots (also called rogue bots) are automated programs that access a website against the operator's wishes or ignore its rules.
- Bingbot
Bingbot is the web crawler of Microsoft Bing.
- DuckDuckBot
DuckDuckBot is the web crawler of the DuckDuckGo search engine.
- Googlebot
Googlebot is the web crawler of Google Search.
- Googlebot Smartphone
Googlebot Smartphone is the mobile version of Googlebot, Google's crawler for Search.
- Google-InspectionTool
Google-InspectionTool is a Google crawler used by testing tools for Search, including the Rich Results Test and the URL Inspection tool in …
- GoogleOther
GoogleOther is a generic crawler from Google that product teams can use to fetch publicly accessible content.
- User agent
The user agent is the software that makes a request to a web server on behalf of a user, and the name of its HTTP request header.
- Web crawler
A web crawler (also called a crawler, spider, or bot) is a program that automatically fetches web pages and follows the links on them to …