Bad bots (also called rogue bots) are automated programs that access a website against the operator’s wishes or ignore its rules. In SEO work they show up in log files and crawl statistics, because they add server load and distort traffic numbers.
What do bad bots do?
Security vendors group them by activity. Akamai lists six common types: DDoS bots, web scraping bots, credential stuffing bots, spam bots, credit card testing bots and click fraud bots. Scalper bots that buy limited stock instantly are another common example. Splunk, citing Imperva’s 2024 Bad Bot Report, puts bot traffic at nearly 50% of internet traffic in 2024, which covers good and bad bots together.
Many bad bots imitate a well-known crawler by sending the user-agent of Googlebot or Bingbot. Google therefore documents how to verify its crawlers. The first method is a reverse DNS lookup on the IP address from your logs, which must resolve to googlebot.com, google.com or googleusercontent.com, followed by a forward lookup that returns the same IP. The second method matches the IP against the JSON files of published Google IP ranges. The user-agent string alone proves nothing.
Robots.txt does not stop bad bots. RFC 9309 states: “These rules are not a form of access authorization.” Compliance is voluntary. Restricting access takes measures on the server or application side, such as rate limiting, authentication or bot management.
Example
A log file shows 40,000 requests per hour with the user-agent Googlebot/2.1. Reverse DNS for the IP does not end in googlebot.com or google.com. The traffic comes from a fake Googlebot and can be blocked without affecting real crawling. IP addresses in log files can be personal data (CJEU, judgment of 19 October 2016, C-582/14, Breyer).
Common mistakes
- Trusting the user agent. The user-agent string can be set freely. If someone falsely claims to be Google, Google points to reverse DNS checks and its published IP ranges.
- Using robots.txt as protection. RFC 9309 states that its rules are not a form of access authorization.
FAQ
What are good bots and bad bots? Good bots, such as search engine crawlers, follow your rules and serve a purpose you accept. Bad bots scrape, attack or abuse a site.
Is being a bot illegal? Site operators set their own rules for bots, for example in robots.txt and in their terms of use. RFC 9309 states that robots.txt rules are not a form of access authorization.
This is not legal advice.
Sources
- Verifying Googlebot and other Google crawlers – Google Search Centraldevelopers.google.com
- RFC 9309: Robots Exclusion Protocol (IETF)rfc-editor.org
- Bad Bots: 6 Common Bot Attacks – Akamai (secondary)akamai.com
- A Guide to Bots – Splunk (secondary)splunk.com
- CJEU, judgment of 19 October 2016, C-582/14 (Breyer) – EUR-Lexeur-lex.europa.eu
Last updated: