Product Hunt launch · 20% off your first 3 months withSee pricing →
ByteDance logo

ByteDance · training

Is Bytespider blocked on your site?

Bytespider is ByteDance’s training token for ByteDance’s models, and whether it reaches you is decided by one line in your robots.txt. A training crawler with a reputation for crawling aggressively. Blocking it is a common choice and costs little in Western answer engines.

  • Not critical if blocked
  • Bytespider robots.txt token

By · Published · Last updated

What is Bytespider?

Bytespider is the crawler operated by ByteDance, the company behind TikTok and the Doubao assistant. It is widely reported to collect training data for ByteDance’s language models, and site operators often report it among their highest-volume bots.

Bytespider at a glance
OperatorByteDance
Purposetraining · ByteDance’s models
robots.txt tokenBytespider
In the user-agentBytespider; spider-feedback@bytedance.com
CrawlRadar weight1 of the crawler-access weights, never flagged critical

How to allow Bytespider in robots.txt

Give Bytespider its own group with Allow: /. A named group overrides the User-agent: * group, so this works even when the wildcard disallows everything:

User-agent: Bytespider
Allow: /

How to block Bytespider in robots.txt

Replace Allow with Disallow. To block only part of the site, disallow that path instead of /:

User-agent: Bytespider
Disallow: /
  • Tokens are matched case-insensitively, but writing Bytespider exactly as the vendor spells it avoids surprises with stricter parsers.
  • Check your CDN and security plugins too: a firewall rule can block Bytespider without any line in robots.txt.

Should you block Bytespider?

Blocking Bytespider is a common choice and costs little visibility in the answer engines most English-language audiences use. If it is straining your server, a robots.txt rule is the first step; if requests keep coming, a rate limit at your CDN is the reliable one.

How to verify a request is really Bytespider

There is no reliable way: ByteDance publishes no IP ranges for Bytespider. A request claiming to be Bytespider could be anyone, so CrawlRadar records such hits as unverified rather than crediting them to ByteDance.

Other training tokens

ByteDance has no other AI token, but these do the same job for other companies. Each is its own robots.txt line, so deciding on Bytespider decides nothing about them.

Other training tokens
TokenOperatorWhat blocking it costs
GPTBotOpenAIBlocking it keeps your pages out of future OpenAI model training. It does not remove you from ChatGPT Search, which crawls under OAI-SearchBot.
ClaudeBotAnthropicBlocking it keeps your future pages out of Anthropic’s training data. Claude’s search citations come from Claude-SearchBot, which is unaffected.
anthropic-aiAnthropicA legacy Anthropic token. Worth keeping in your robots.txt alongside ClaudeBot so an older group cannot contradict a newer one.
CCBotCommon CrawlCommon Crawl’s archive feeds many models indirectly. Blocking it is a broad training opt-out with a long tail.
AmazonbotAmazonAmazon uses it to improve its products, including Alexa’s answers, and says the content may be used to train its AI models.

Is Bytespider visiting your site?

Your analytics will not tell you. AI crawlers do not run JavaScript, so a tag-based tool such as Google Analytics never records them. CrawlRadar’s collector reads your own request path and records each Bytespider visit, the page it fetched, and whether it came from ByteDance. To check by hand, the guide to tracking AI crawlers has the log commands and each vendor’s IP lists. See the full AI crawler directory for the other tokens CrawlRadar checks.

Frequently asked questions

  • Does Bytespider respect robots.txt?

    Reports from site operators are mixed, and ByteDance publishes no IP ranges to verify requests against. Treat the robots.txt rule as a statement of intent, and enforce it at your CDN if you need certainty.
  • Will blocking Bytespider affect my TikTok account?

    No. The rule controls whether ByteDance can crawl your website. Your TikTok account and any links you share there are unaffected.

See whether Bytespider actually visits you

robots.txt says who may crawl; only your own request log says who did. CrawlRadar’s collector records every AI crawler visit, verifies it against the vendor’s published ranges, and shows the pages each one fetched.

14-day free trial.No credit card.

Or start with the free AI crawler checker.