# YouBot

Category: [AI Bots & Crawlers](https://seo-geo-glossary.com/en/glossary/category/ai-bots-crawlers/) › [Other AI bots](https://seo-geo-glossary.com/en/glossary/category/ai-bots-crawlers/other-ai-bots/)

**YouBot** is the web crawler of the You.com search engine. This entry is about the bot, not about the KUKA youBot research robot or other products with the same name. YouBot discovers and indexes web pages, and You.com uses that index for its search results.

## How does YouBot work?

You.com describes YouBot as a crawler that automatically discovers and indexes pages. Its user agent contains `YouBot/1.0` and the token `env:prod`. According to You.com, legitimate requests come from the IP range `68.67.112.0/24`. For crawling problems the documentation lists the contact address spider@you.com.

Because the user agent and the IP range are published, you can check a log entry against both. A request that claims to be YouBot but comes from another range does not match the documentation.

## How can you control YouBot?

You.com states that YouBot respects robots.txt rules for its user agent, including crawl-delay. It caches the robots.txt file for 30 minutes, so a change can take up to half an hour to take effect.

## Example

```robots
User-agent: YouBot
Crawl-delay: 10
Disallow: /internal/
```

The rule sets a `Crawl-delay` of 10 for the crawler and excludes the `/internal/` directory.

## Common mistakes

- **Expecting changes right away.** According to You.com, YouBot caches robots.txt for 30 minutes, so a change can take up to half an hour.
- **Trusting every request that uses the YouBot name.** You.com names the IP range `68.67.112.0/24` and reverse DNS names following the pattern `youbot-<IP>.search.you.com`. A request that claims to be YouBot but comes from elsewhere does not match the documentation.

## FAQ

**Is YouBot an AI crawler?**
YouBot feeds the You.com search engine. You.com also offers AI-generated answers, but the documentation describes the crawler's purpose as search and makes no statement about model training.

**How fast does a robots.txt change apply?**
According to You.com, within 30 minutes at most.

## At a glance

- **Operator:** You.com
- **Purpose:** Search (discovering and indexing pages)
- **robots.txt token:** YouBot
- **Respects robots.txt:** yes, including Crawl-delay; robots.txt is cached for 30 minutes
- **IP list:** IP range 68.67.112.0/24
- **Docs:** [You.com: YouBot](https://you.com/docs/youbot)

Last checked: October 8, 2026

## AI bots compared

| Bot | Operator | Purpose | robots.txt token | robots.txt according to the operator | IP list published? |
|---|---|---|---|---|---|
| Google-Extended | Google | Controls Gemini training and grounding (no crawler of its own) | Google-Extended | robots.txt token only; fetching is done by existing Google crawlers | n/a (no user agent of its own) |
| GPTBot | OpenAI | Training | GPTBot | Disallowing it signals content should not be used to train foundation models | yes (openai.com/gptbot.json) |
| OAI-SearchBot | OpenAI | Search (ChatGPT) | OAI-SearchBot | opted-out sites are not shown in ChatGPT search answers | yes (openai.com/searchbot.json) |
| ChatGPT-User | OpenAI | User-triggered fetch | ChatGPT-User | robots.txt rules may not apply, says OpenAI (user-initiated) | yes (openai.com/chatgpt-user.json) |
| ClaudeBot | Anthropic | Training | ClaudeBot | yes, Anthropic says its bots honor robots.txt | yes (claude.com/crawling/bots.json) |
| Claude-SearchBot | Anthropic | Search (Claude) | Claude-SearchBot | yes, Anthropic says its bots honor robots.txt | yes (claude.com/crawling/bots.json) |
| PerplexityBot | Perplexity | Search (Perplexity), not used for model training | PerplexityBot | controllable via robots.txt; the docs contain no explicit compliance statement | yes (perplexity.com/perplexitybot.json) |
| Applebot-Extended | Apple | Controls AI training (does not crawl, says Apple) | Applebot-Extended | robots.txt token only, applies to content Applebot already crawled | n/a (no crawler of its own) |
| CCBot | Common Crawl | Open web crawl archive | CCBot | can be blocked via robots.txt, says Common Crawl | yes (index.commoncrawl.org/ccbot.json) |
| Meta-ExternalAgent | Meta | Training foundation AI models, improving products | meta-externalagent | can be blocked via robots.txt, says Meta | not stated by the operator |

Sources: [OpenAI: Overview of OpenAI Crawlers](https://developers.openai.com/api/docs/bots); [Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler? (updated 2026-04-07)](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler); [Perplexity: Perplexity Crawlers](https://docs.perplexity.ai/guides/bots); [Google: Google's common crawlers (last updated 2026-07-14)](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers); [Google: Overview of Google crawlers and fetchers (last updated 2026-06-12)](https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers); [Google: Verifying Google crawlers and fetchers (IP range files)](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests); [Apple: About Applebot (2026-09-04)](https://support.apple.com/en-us/119829); [Meta: Meta Web Crawlers](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/); [Common Crawl: CCBot](https://commoncrawl.org/ccbot). Last checked: October 8, 2026

## Related terms

- [AI crawler](https://seo-geo-glossary.com/en/glossary/ai-crawler/index.md)
- [robots.txt](https://seo-geo-glossary.com/en/glossary/robots-txt/index.md)
- [Crawl-delay](https://seo-geo-glossary.com/en/glossary/crawl-delay/index.md)
- [User agent](https://seo-geo-glossary.com/en/glossary/user-agent/index.md)

## Sources

- [YouBot – You.com Docs (as of 2026-10-07)](https://you.com/docs/youbot)

---

Last updated: October 8, 2026
URL: https://seo-geo-glossary.com/en/glossary/youbot/