Bot traffic is any traffic to a website that comes from automated programs instead of people. It includes wanted visitors such as search engine crawlers and monitoring services, and unwanted ones such as scrapers, credential-stuffing tools and DDoS bots.
Good and bad bots
Search engine crawlers fetch pages so they can be indexed. Uptime and monitoring services send requests to check availability. Other bots copy content, probe logins or overload servers. A robots.txt rule only reaches bots that choose to follow it.
Where does bot traffic show up?
The access log of the server is the most complete record, because every request ends up there. Analytics tools usually capture only part of it (assessment). Google Analytics 4 excludes traffic from known bots and spiders automatically, based on Google research and the IAB’s International Spiders and Bots List. Google says this exclusion cannot be turned off and that the amount excluded is not shown. Bots not on the list can still appear as visitors, often as unexplained spikes.
Example
A shop sees a jump in sessions every Tuesday, with very short visit durations. The server log shows one user agent requesting thousands of product pages in that window. The example is invented.
Common questions
How much of the internet is bots? Published shares differ. Vercel’s documentation says bots generate nearly half of internet traffic, Akamai says up to 70% of website traffic. The numbers depend on the source and on what counts as a bot.
More on the searchVIU blog
Sources
- Known bot-traffic exclusion - Google Analytics Helpsupport.google.com
- Bot Management - Vercel Docsvercel.com
- What are traffic bots - Akamai Glossaryakamai.com
- Verifying Googlebot and other Google crawlers - Google Search Centraldevelopers.google.com
Last updated: