cohere-ai is a user agent token attributed to the AI company Cohere. It shows up in server logs and bot directories. As far as research showed, Cohere has not published its own documentation for it.
At a glance
- Operator
- Cohere (per Known Agents, not confirmed by Cohere)
- Purpose
- not stated by Cohere
- robots.txt token
cohere-ai- Respects robots.txt
- no statement from Cohere; Known Agents expects compliance (secondary source)
- Docs
- no operator documentation found
Last checked:
What is known about cohere-ai?
The directory Known Agents lists cohere-ai as an unconfirmed agent. According to reports, it is possibly dispatched by Cohere’s AI chat products when they need to retrieve content from the internet in response to a user prompt. Cohere has not commented on its purpose. Training use, IP ranges and the full user agent string are not documented by Cohere.
Whether a bot follows robots.txt is up to its operator. There is no official statement from Cohere for cohere-ai.
| Bot | Operator | Purpose | robots.txt token | robots.txt according to the operator | IP list published? |
|---|---|---|---|---|---|
| Google-Extended | Controls Gemini training and grounding (no crawler of its own) | Google-Extended | robots.txt token only; fetching is done by existing Google crawlers | n/a (no user agent of its own) | |
| GPTBot | OpenAI | Training | GPTBot | Disallowing it signals content should not be used to train foundation models | yes (openai.) |
| OAI-SearchBot | OpenAI | Search (ChatGPT) | OAI-SearchBot | opted-out sites are not shown in ChatGPT search answers | yes (openai.) |
| ChatGPT-User | OpenAI | User-triggered fetch | ChatGPT-User | robots.txt rules may not apply, says OpenAI (user-initiated) | yes (openai.) |
| ClaudeBot | Anthropic | Training | ClaudeBot | yes, Anthropic says its bots honor robots.txt | yes (claude.) |
| Claude-SearchBot | Anthropic | Search (Claude) | Claude-SearchBot | yes, Anthropic says its bots honor robots.txt | yes (claude.) |
| PerplexityBot | Perplexity | Search (Perplexity), not used for model training | PerplexityBot | controllable via robots.txt; the docs contain no explicit compliance statement | yes (perplexity.) |
| Applebot-Extended | Apple | Controls AI training (does not crawl, says Apple) | Applebot-Extended | robots.txt token only, applies to content Applebot already crawled | n/a (no crawler of its own) |
| CCBot | Common Crawl | Open web crawl archive | CCBot | can be blocked via robots.txt, says Common Crawl | yes (index.) |
| Meta-ExternalAgent | Meta | Training foundation AI models, improving products | meta-externalagent | can be blocked via robots.txt, says Meta | not stated by the operator |
A selection of ten AI bots from major operators, not a complete list. All values come from each operator's own documentation (as of 8 October 2026). The table does not show whether a bot actually complies with robots.txt; it only reflects what operators state. User agents can be spoofed; published IP lists or reverse DNS identify genuine requests. Sources: OpenAI: Overview of OpenAI Crawlers; Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler? (updated 2026-04-07); Perplexity: Perplexity Crawlers; Google: Google's common crawlers (last updated 2026-07-14); Google: Overview of Google crawlers and fetchers (last updated 2026-06-12); Google: Verifying Google crawlers and fetchers (IP range files); Apple: About Applebot (2026-09-04); Meta: Meta Web Crawlers; Common Crawl: CCBot. Last checked: .
How can you control cohere-ai?
Under RFC 9309, a robots.txt group applies to crawlers whose user agent token matches the group. A rule for cohere-ai takes effect if the bot uses that token and honors the rule. Whether that is the case can only be checked in your own logs.
Example
User-agent: cohere-ai
Disallow: /
To check the effect, search the logs for requests with the token and see whether they stop after the change.
FAQ
Does Cohere run a crawler for training data? There is no documented statement from Cohere. Directories describe cohere-ai as fetching on user prompts.
How do I spot cohere-ai in my logs?
By the token cohere-ai in the user agent. Because that value can be faked, it does not prove the request came from Cohere.
Sources
- cohere-ai – Known Agents (secondary)knownagents.com
- RFC 9309: Robots Exclusion Protocol – IETFrfc-editor.org
Last updated: