Agent analytics, in an SEO context, is the analysis of visits by AI bots and AI agents to a website, usually based on server or CDN logs. The term is not standardized and comes mainly from vendors of such tools. Product analytics vendors such as Amplitude and Pendo use the same words for something else, namely measuring how users interact with an AI agent inside a product.
At a glance
- Type
- Analysis (method, not standardized)
- Affects
- Visits by AI bots and AI agents
- Used in
- Server, proxy and CDN logs
Last checked:
What does agent analytics measure?
Classic web analytics runs in the visitor’s browser. Profound says it uses server logs instead of JavaScript trackers, and Siteline states that most AI bots bypass JavaScript. Requests from AI bots appear in the server or CDN log whether or not a tracking script loads.
Typical questions are which bots visit the site, which pages they fetch, which status codes they receive and how this changes over time. Cloudflare groups bot behavior into search, agent and training. A search bot collects content to answer questions later, an agent bot acts in real time for a person, and a training bot crawls content to train a model.
How does it relate to log file analysis?
Agent analytics is log file analysis with a focus on AI bots (assessment). The data source stays the same. What changes is the mapping of user agents to operators, purpose and page types.
IP addresses in logs can be personal data (CJEU, judgment of 19 October 2016, C-582/14, Breyer).
Example
This command counts the status codes that GPTBot received in a log in Combined Log Format.
grep "GPTBot" access.log | awk '{print $9}' | sort | uniq -c
A high number of 404 responses means the bot requests URLs that no longer exist.
Common mistakes
- Relying on analytics data only. According to Siteline, most AI bots bypass JavaScript. A tracking script does not record these visits, while the server or CDN log does.
- Trusting the user agent alone. Requests can claim to come from a known crawler. Google therefore describes verification through reverse DNS and published IP ranges, and other operators publish IP lists.
- Lumping all AI bots together. Cloudflare separates search, agent and training by behavior. The groups show different things, such as fetches for answers or for training data.
FAQ
Why is Google Analytics not enough for this? According to Siteline, most AI bots bypass JavaScript, so a tracking script is not triggered.
Which data do I need? Access to server, proxy or CDN logs that include user agent, status code and requested URL.
More on the searchVIU blog
Sources
- Agent Analytics – Profound (vendor page)tryprofound.com
- Bots and AI classification – Cloudflare Docsdevelopers.cloudflare.com
- Why We Created Agent Analytics – Amplitude (other meaning)amplitude.com
- Overview of Agent Analytics – Pendo Support (other meaning)support.pendo.io
- Why track AI bot & agent traffic? – Siteline Docs (vendor documentation)docs.siteline.ai
- Verifying Google crawlers and fetchers – Google Search Centraldevelopers.google.com
- CJEU, judgment of 19 October 2016, C-582/14 (Breyer) – EUR-Lexeur-lex.europa.eu
Last updated: