Applebot-Extended is not a crawler. It is a robots.txt control that tells Apple not to use Applebot-crawled content to train its general-purpose foundation models. Blocking it does not remove pages from Apple search results.
User-agent: Applebot Allow: / User-agent: Applebot-Extended Disallow: /
Applebot-Extended is a secondary robots.txt user-agent token that gives publishers a separate choice about model-training use. Apple says it can be disallowed when you do not want website content used to train the general-purpose foundation models powering generative AI features across Apple products, including Apple Intelligence, Services, and Developer Tools.
Applebot-Extended does not crawl webpages itself. The actual web crawl is performed by Applebot. The Extended token is consulted later to determine how Apple may use the data Applebot collected.
The most common selective policy is to allow Applebot and disallow Applebot-Extended. This keeps eligible pages discoverable through Apple search experiences such as Siri, Spotlight, and Safari while expressing an opt-out from the foundation-model training use covered by Applebot-Extended.
Apple explicitly states that pages disallowing Applebot-Extended can still appear in search results. Therefore, do not block Applebot itself unless you also want to restrict Apple search crawling.
Apple documents another, separate use of crawled content: providing additional context and up-to-date information when models generate answers for display in Apple products and services. Apple says publishers can use the nosnippet meta tag on specific content to opt out of use in broad world-knowledge answers. That control is different from Applebot-Extended, which covers training of general-purpose foundation models.
For paywalled pages, Apple also supports the isAccessibleForFree: false structured-data property. These controls solve different problems and should not be treated as interchangeable.
Use Disallow: / to opt the entire host out of the covered training use. Use directory-specific rules when only selected sections should be excluded. Robots.txt is scoped per host, protocol, and port, so each relevant subdomain needs its own file.
Review the live file after every deployment or platform migration. A robots.txt rule stored in a CMS field has no effect unless the public host actually serves it from the expected root URL with the intended content. Also confirm that a CDN, security plugin, or stale cache is not serving an older version. When only selected directories are excluded, test both an allowed public URL and a disallowed URL so the policy is not broader than intended.
For a broader multi-company policy, start with the allow search, block AI training template and adapt each user agent separately.
No. Apple says Applebot-Extended does not crawl webpages; Applebot performs the crawl and the Extended token controls how that data may be used.
No. Apple states that pages disallowing Applebot-Extended can still appear in Apple search results.
It controls whether Applebot-crawled website content may be used to train Apple’s general-purpose foundation models for generative AI features.
No. Applebot-Extended addresses foundation-model training, while Apple documents nosnippet as a separate control for use of page content in broad world-knowledge answers.
Yes. Use path-specific Disallow directives under the Applebot-Extended group instead of Disallow: / for the whole host.
Applebot robots.txt Rules for Apple SearchApplebot is Apple’s search crawler for experiences including Siri, Spotlight, and Safari. It follows Applebot-specific robots.txt rules, falls back to Googlebot rules when no Applebot group exists, and does not support Crawl-delay.
Allow Search, Block AI Training robots.txt TemplateUse separate user-agent groups: block documented training or dataset agents such as GPTBot and ClaudeBot, while allowing ordinary search and dedicated AI-search crawlers.
How to Block AI Crawlers with robots.txtTo block an AI crawler, add a group for its exact User-agent token and use Disallow: /. Block only the crawler purposes you intend to restrict, keep ordinary search crawlers open when visibility matters, and remember that robots.txt is a request to compliant bots rather than a security control.
robots.txt for AI Training: Build a Selective Opt-OutTo restrict AI training access, identify each provider’s exact training or dataset token and disallow it in a dedicated robots.txt group. Keep search crawlers in separate allowed groups, treat product-control tokens such as Google-Extended according to their documentation, and use authentication or server-side controls when access must be enforced.
Does robots.txt Stop AI Scraping? Limits and Better ControlsRobots.txt can stop compliant AI crawlers from requesting selected URLs, but it cannot enforce access against unidentified scrapers, spoofed user agents, browsers, or tools that ignore the protocol. Use precise crawler rules for policy, then add authentication, authorization, rate limits, or firewall controls where access must actually be prevented.
