By Iliass El Barhoumi · Published · Last updated
What is Amazonbot?
Amazonbot is Amazon’s web crawler. Amazon says it is used to improve its products and services, helping provide more accurate information to customers, and that the content may be used to train Amazon AI models. Amazon runs search under a separate token, Amzn-SearchBot, and user-requested fetches under Amzn-User.
| Operator | Amazon |
|---|---|
| Purpose | training · Alexa and Amazon’s AI models |
| robots.txt token | Amazonbot |
| In the user-agent | compatible; Amazonbot/0.1 |
| CrawlRadar weight | 1 of the crawler-access weights, never flagged critical |
| Vendor docs | Amazon: About Amazonbot |
How to allow Amazonbot in robots.txt
Give Amazonbot its own group with Allow: /. A named group overrides the User-agent: * group, so this works even when the wildcard disallows everything:
User-agent: Amazonbot
Allow: /How to block Amazonbot in robots.txt
Replace Allow with Disallow. To block only part of the site, disallow that path instead of /:
User-agent: Amazonbot
Disallow: /- Tokens are matched case-insensitively, but writing
Amazonbotexactly as the vendor spells it avoids surprises with stricter parsers. - Check your CDN and security plugins too: a firewall rule can block Amazonbot without any line in robots.txt.
Should you block Amazonbot?
Allow Amazonbot if Alexa and Amazon’s assistants are a channel your customers use, and block it if you want out of Amazon’s model training. Amazon documents each of its tokens as independent, so blocking Amazonbot does not block Amzn-SearchBot, and says changes take about 24 hours to apply.
How to verify a request is really Amazonbot
Run a reverse DNS lookup on the requesting IP: a genuine Amazonbot host resolves under crawl.amazonbot.amazon, and a forward lookup on that hostname returns the same IP. Amazon also publishes its addresses at https://developer.amazon.com/amazonbot/ip-addresses/. CrawlRadar’s collector runs this check on every hit.
Other training tokens
Amazon has no other AI token, but these do the same job for other companies. Each is its own robots.txt line, so deciding on Amazonbot decides nothing about them.
| Token | Operator | What blocking it costs |
|---|---|---|
GPTBot | OpenAI | Blocking it keeps your pages out of future OpenAI model training. It does not remove you from ChatGPT Search, which crawls under OAI-SearchBot. |
ClaudeBot | Anthropic | Blocking it keeps your future pages out of Anthropic’s training data. Claude’s search citations come from Claude-SearchBot, which is unaffected. |
anthropic-ai | Anthropic | A legacy Anthropic token. Worth keeping in your robots.txt alongside ClaudeBot so an older group cannot contradict a newer one. |
CCBot | Common Crawl | Common Crawl’s archive feeds many models indirectly. Blocking it is a broad training opt-out with a long tail. |
Bytespider | ByteDance | A training crawler with a reputation for crawling aggressively. Blocking it is a common choice and costs little in Western answer engines. |
Is Amazonbot visiting your site?
Your analytics will not tell you. AI crawlers do not run JavaScript, so a tag-based tool such as Google Analytics never records them. CrawlRadar’s collector reads your own request path and records each Amazonbot visit, the page it fetched, and whether it came from Amazon. To check by hand, the guide to tracking AI crawlers has the log commands and each vendor’s IP lists. See the full AI crawler directory for the other tokens CrawlRadar checks.
Frequently asked questions
How do I verify that a request is really Amazonbot?
Amazon publishes Amazonbot’s IP addresses at developer.amazon.com/amazonbot/ip-addresses, and genuine hosts resolve under crawl.amazonbot.amazon on a reverse DNS lookup. CrawlRadar’s collector runs the reverse-DNS check on every hit.Does blocking Amazonbot affect my Amazon seller listings?
No. The rule controls crawling of your own website. Listings on Amazon are not fetched from your site by Amazonbot.