By Iliass El Barhoumi · Published · Last updated
What is Bytespider?
Bytespider is the crawler operated by ByteDance, the company behind TikTok and the Doubao assistant. It is widely reported to collect training data for ByteDance’s language models, and site operators often report it among their highest-volume bots.
| Operator | ByteDance |
|---|---|
| Purpose | training · ByteDance’s models |
| robots.txt token | Bytespider |
| In the user-agent | Bytespider; spider-feedback@bytedance.com |
| CrawlRadar weight | 1 of the crawler-access weights, never flagged critical |
How to allow Bytespider in robots.txt
Give Bytespider its own group with Allow: /. A named group overrides the User-agent: * group, so this works even when the wildcard disallows everything:
User-agent: Bytespider
Allow: /How to block Bytespider in robots.txt
Replace Allow with Disallow. To block only part of the site, disallow that path instead of /:
User-agent: Bytespider
Disallow: /- Tokens are matched case-insensitively, but writing
Bytespiderexactly as the vendor spells it avoids surprises with stricter parsers. - Check your CDN and security plugins too: a firewall rule can block Bytespider without any line in robots.txt.
Should you block Bytespider?
Blocking Bytespider is a common choice and costs little visibility in the answer engines most English-language audiences use. If it is straining your server, a robots.txt rule is the first step; if requests keep coming, a rate limit at your CDN is the reliable one.
How to verify a request is really Bytespider
There is no reliable way: ByteDance publishes no IP ranges for Bytespider. A request claiming to be Bytespider could be anyone, so CrawlRadar records such hits as unverified rather than crediting them to ByteDance.
Is Bytespider visiting your site?
Your analytics will not tell you. AI crawlers do not run JavaScript, so a tag-based tool such as Google Analytics never records them. CrawlRadar’s collector reads your own request path and records each Bytespider visit, the page it fetched, and whether it came from ByteDance. See the full AI crawler directory for the other tokens CrawlRadar checks.
Frequently asked questions
Does Bytespider respect robots.txt?
Reports from site operators are mixed, and ByteDance publishes no IP ranges to verify requests against. Treat the robots.txt rule as a statement of intent, and enforce it at your CDN if you need certainty.
Will blocking Bytespider affect my TikTok account?
No. The rule controls whether ByteDance can crawl your website. Your TikTok account and any links you share there are unaffected.