AI Bots & Crawlers

An AI crawler is an automated program that fetches web pages for AI applications.

An AI search crawler is a crawler that fetches web pages for the search feature of an AI product, so the product can surface and link them …

A browser agent is an AI system that operates a web browser the way a person does.

A training crawler is an AI crawler that fetches web pages to collect content for training AI models.

A user-triggered fetcher is an automated program that loads a web page because a person asked a product to do so.

A web scraper is a program that automatically extracts data from web pages and converts it into a structured format such as a spreadsheet …

anthropic-ai is a user agent name that appears in many robots.txt files to keep Anthropic's crawlers out.

ChatGPT agent is the OpenAI feature that lets ChatGPT carry out multi-step tasks on the web for a user.

ChatGPT-User is an OpenAI user agent that fetches web pages when someone using ChatGPT or a Custom GPT triggers an action.

ClaudeBot is the web crawler of Anthropic, the company behind the AI assistant Claude.

Claude-SearchBot is an Anthropic crawler that navigates the web to improve the quality of search results for Claude users.

Claude-User is the user agent Claude, Anthropic's AI assistant, uses to access websites when a person asks Claude a question.

GPTBot is OpenAI's web crawler for collecting publicly available content that may be used to train its generative AI models.

OAI-SearchBot is a crawler run by OpenAI.

PerplexityBot is the crawler of the AI search engine Perplexity.

Perplexity-User is a Perplexity user agent that visits web pages when a person asks Perplexity a question and the answer needs a page.

Amazonbot is Amazon's web crawler.

Amzn-SearchBot is an Amazon crawler that, according to Amazon, improves search experiences in Amazon products and services.

Amzn-User is an Amazon fetcher that loads web pages on behalf of users.

Applebot is Apple's web crawler.

Bytespider is a web crawler attributed to ByteDance, the company behind TikTok.

CCBot is the web crawler of Common Crawl, a non-profit that maintains an open repository of web crawl data.

cohere-ai is a user agent token attributed to the AI company Cohere.

Diffbot is a company that uses machine learning to turn web pages into structured data.

DuckAssistBot is the web crawler DuckDuckGo uses for the AI-assisted answers in DuckDuckGo Search.

Google-CloudVertexBot is a Google crawler that fetches a website when the site owner has requested it, to build Vertex AI agents.

Meta-ExternalAgent is a web crawler run by Meta, the company behind Facebook, Instagram and WhatsApp.

Meta-ExternalFetcher is a Meta user agent that fetches individual links when a user asks for them.

Meta-WebIndexer is a Meta crawler that, according to Meta, navigates the web to improve the quality of search results in Meta AI.

MistralAI-User is a user agent from Mistral AI that visits a web page when a person asks Mistral's assistant a question.

YouBot is the web crawler of the You.com search engine.

Agent analytics, in an SEO context, is the analysis of visits by AI bots and AI agents to a website, usually based on server or CDN logs.

Blocking AI bots means denying specific AI crawlers and fetchers access to a website.

The crawl-to-refer ratio (also written crawl-to-referral ratio) compares how many pages a platform crawls with how often it sends users to …

Pay per crawl is a model in which a crawler pays for each page it fetches.

Web Bot Auth is a method for automated clients such as crawlers and AI agents to identify themselves to websites with a cryptographic …