- Amazonbot
Amazonbot is Amazon's web crawler.
- Amzn-SearchBot
Amzn-SearchBot is an Amazon crawler that, according to Amazon, improves search experiences in Amazon products and services.
- Amzn-User
Amzn-User is an Amazon fetcher that loads web pages on behalf of users.
- Applebot / Applebot-Extended
Applebot is Apple's web crawler.
- Bytespider
Bytespider is a web crawler attributed to ByteDance, the company behind TikTok.
- CCBot
CCBot is the web crawler of Common Crawl, a non-profit that maintains an open repository of web crawl data.
- cohere-ai
cohere-ai is a user agent token attributed to the AI company Cohere.
- Diffbot
Diffbot is a company that uses machine learning to turn web pages into structured data.
- DuckAssistBot
DuckAssistBot is the web crawler DuckDuckGo uses for the AI-assisted answers in DuckDuckGo Search.
- Google-CloudVertexBot
Google-CloudVertexBot is a Google crawler that fetches a website when the site owner has requested it, to build Vertex AI agents.
- Meta-ExternalAgent
Meta-ExternalAgent is a web crawler run by Meta, the company behind Facebook, Instagram and WhatsApp.
- Meta-ExternalFetcher
Meta-ExternalFetcher is a Meta user agent that fetches individual links when a user asks for them.
- Meta-WebIndexer
Meta-WebIndexer is a Meta crawler that, according to Meta, navigates the web to improve the quality of search results in Meta AI.
- MistralAI-User
MistralAI-User is a user agent from Mistral AI that visits a web page when a person asks Mistral's assistant a question.
- YouBot
YouBot is the web crawler of the You.com search engine.