By Iliass El Barhoumi · Published · Last updated
What is Applebot-Extended?
Applebot-Extended is a robots.txt token, not a crawler. Apple crawls with Applebot, which powers search in Spotlight, Siri and Safari, and reads Applebot-Extended to decide whether that content may be used to train Apple’s generative foundation models, including those behind Apple Intelligence.
| Operator | Apple |
|---|---|
| Purpose | policy directive · Apple Intelligence training |
| robots.txt token | Applebot-Extended |
| In the user-agent | Never sent: this token is only read from robots.txt |
| CrawlRadar weight | 1 of the crawler-access weights, never flagged critical |
| Vendor docs | Apple: About Applebot |
How to allow Applebot-Extended in robots.txt
Give Applebot-Extended its own group with Allow: /. A named group overrides the User-agent: * group, so this works even when the wildcard disallows everything:
User-agent: Applebot-Extended
Allow: /How to block Applebot-Extended in robots.txt
Replace Allow with Disallow. To block only part of the site, disallow that path instead of /:
User-agent: Applebot-Extended
Disallow: /- Tokens are matched case-insensitively, but writing
Applebot-Extendedexactly as the vendor spells it avoids surprises with stricter parsers. - Check your CDN and security plugins too: a firewall rule can block Applebot-Extended without any line in robots.txt.
Should you block Applebot-Extended?
Disallowing Applebot-Extended opts you out of Apple’s model training while keeping you in Spotlight, Siri and Safari results, which Apple says remain unaffected. Apple also notes that crawled content can add context to AI answers in Siri and Search; the opt-out for that is the nosnippet meta tag, not this token.
How to verify a request is really Applebot-Extended
You cannot, because no request ever carries the Applebot-Extended name. Apple fetches pages with its regular crawlers and reads Applebot-Extended from robots.txt as a permission. A log line claiming to be Applebot-Extended is not from Apple.
Is Applebot-Extended visiting your site?
Your analytics will not tell you. AI crawlers do not run JavaScript, so a tag-based tool such as Google Analytics never records them. Applebot-Extended itself never visits, but Apple’s crawlers do, and CrawlRadar’s collector records them from your own server. See the full AI crawler directory for the other tokens CrawlRadar checks.
Frequently asked questions
Does blocking Applebot-Extended remove me from Siri or Spotlight?
No. Siri, Spotlight and Safari search use Applebot. Apple states that content stays discoverable in those products even when Applebot-Extended is disallowed.
Why is there no Applebot-Extended in my server logs?
Nothing crawls under that name. Requests come from Applebot, and the token is read from robots.txt as a permission about how Applebot’s crawl may be used.