Bytespider is a web crawler attributed to ByteDance, the company behind TikTok. It appears in server logs under the user agent token Bytespider. ByteDance has no official documentation for it that could be found, so the attribution rests on press reports and bot directories.
At a glance
- Operator
- ByteDance (attribution according to reports)
- Purpose
- according to Fortune, scraping web data, likely for AI training and TikTok search
- robots.txt token
Bytespider- Respects robots.txt
- no statement from the operator
- Docs
- no operator documentation found
Last checked:
What is known about Bytespider?
Fortune reported in October 2024 that Bytespider was scraping web data, likely to train generative AI models and to improve search in TikTok. Fortune also reported that the bot did not respect robots.txt and operated at far higher request rates than other AI crawlers. Servebolt, a hosting company, reported millions of requests per day from Bytespider and blocked the bot. These are third-party observations from that time. ByteDance has not published a purpose, IP ranges or a way to verify genuine requests, as far as research showed.
For site owners this means a visit under this name cannot be tied to an operator with certainty. A user agent string can be set to anything.
| Bot | Operator | Purpose | robots.txt token | robots.txt according to the operator | IP list published? |
|---|---|---|---|---|---|
| Google-Extended | Controls Gemini training and grounding (no crawler of its own) | Google-Extended | robots.txt token only; fetching is done by existing Google crawlers | n/a (no user agent of its own) | |
| GPTBot | OpenAI | Training | GPTBot | Disallowing it signals content should not be used to train foundation models | yes (openai.) |
| OAI-SearchBot | OpenAI | Search (ChatGPT) | OAI-SearchBot | opted-out sites are not shown in ChatGPT search answers | yes (openai.) |
| ChatGPT-User | OpenAI | User-triggered fetch | ChatGPT-User | robots.txt rules may not apply, says OpenAI (user-initiated) | yes (openai.) |
| ClaudeBot | Anthropic | Training | ClaudeBot | yes, Anthropic says its bots honor robots.txt | yes (claude.) |
| Claude-SearchBot | Anthropic | Search (Claude) | Claude-SearchBot | yes, Anthropic says its bots honor robots.txt | yes (claude.) |
| PerplexityBot | Perplexity | Search (Perplexity), not used for model training | PerplexityBot | controllable via robots.txt; the docs contain no explicit compliance statement | yes (perplexity.) |
| Applebot-Extended | Apple | Controls AI training (does not crawl, says Apple) | Applebot-Extended | robots.txt token only, applies to content Applebot already crawled | n/a (no crawler of its own) |
| CCBot | Common Crawl | Open web crawl archive | CCBot | can be blocked via robots.txt, says Common Crawl | yes (index.) |
| Meta-ExternalAgent | Meta | Training foundation AI models, improving products | meta-externalagent | can be blocked via robots.txt, says Meta | not stated by the operator |
A selection of ten AI bots from major operators, not a complete list. All values come from each operator's own documentation (as of 8 October 2026). The table does not show whether a bot actually complies with robots.txt; it only reflects what operators state. User agents can be spoofed; published IP lists or reverse DNS identify genuine requests. Sources: OpenAI: Overview of OpenAI Crawlers; Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler? (updated 2026-04-07); Perplexity: Perplexity Crawlers; Google: Google's common crawlers (last updated 2026-07-14); Google: Overview of Google crawlers and fetchers (last updated 2026-06-12); Google: Verifying Google crawlers and fetchers (IP range files); Apple: About Applebot (2026-09-04); Meta: Meta Web Crawlers; Common Crawl: CCBot. Last checked: .
How can you control Bytespider?
Under RFC 9309, robots.txt rules apply to the user agent a crawler uses. An entry for Bytespider works if the bot identifies itself with that name and follows the rule. RFC 9309 also states that these rules are not a form of access authorization. Anyone who needs a dependable block should add server-side measures (an assessment).
Example
User-agent: Bytespider
Disallow: /
After the change, the log file shows whether requests with this user agent keep arriving.
Common mistakes
- Treating robots.txt as access protection. RFC 9309 states that the rules are not a form of access authorization. Anyone who must reliably prevent access needs server-side measures.
- Trusting the user agent as proof of origin. The value comes from the request and can be set to anything. ByteDance has published neither IP ranges nor a verification method.
FAQ
Does Bytespider definitely belong to ByteDance? The attribution is documented by third parties only. ByteDance has not confirmed it.
Does Bytespider follow robots.txt? There is no verified statement from the operator. Press reports say it did not. The only reliable evidence is your own log.
Sources
Last updated: