Applebot-Extended: Control Apple AI Training Use

Applebot-Extended is not a crawler that fetches pages. Apple uses this robots.txt user-agent as a data-use signal for content collected by Applebot, allowing publishers to restrict use of that content for training Apple’s generative foundation models while keeping Applebot search crawling available.

Independent technical reference Updated Editorial approach Report a correction
Applebot-crawled content remaining available for search while a separate model-training route is blocked

Primary reference

Official crawler documentation

Use the provider's current documentation to confirm the crawler token and purpose before changing a production robots.txt file.

Open Applebot documentation Reference checked July 6, 2026

Copy-paste robots.txt example

User-agent: Applebot
Allow: /

User-agent: Applebot-Extended
Disallow: /

Applebot-Extended is a data-use control, not a second Apple crawler. Apple explicitly states that Applebot-Extended does not crawl web pages. Its purpose is to determine how content collected by the Applebot crawler may be used.

How to verify an Applebot-Extended opt-out without looking for visits

Do not look for Applebot-Extended requests in your server log as evidence of compliance: Apple says this token does not crawl pages. Instead, verify the policy in the public robots.txt response:

Review the complete Applebot-Extended example in the next section, then check the published robots.txt against it.

  1. Fetch https://your-domain.example/robots.txt on the exact public hostname.
  2. Check that the Applebot-Extended group is present and that its Disallow line covers the intended paths.
  3. Check Applebot separately if normal Apple discovery should remain possible.
  4. Make sure the CMS or CDN has not substituted an older robots.txt file.

The opt-out is a downstream-use instruction for material collected by Applebot, not an encryption method, a way to delete prior training data, or a promise that a page will vanish from Apple search. If content must not be read by unauthorized parties, require authentication. Apple's official explanation explicitly distinguishes the crawler from the training-use control.

What blocking Applebot-Extended changes

Disallowing Applebot-Extended tells Apple not to use the affected site content to train the general-purpose generative foundation models described in Apple’s documentation.

User-agent: Applebot-Extended
Disallow: /

This rule does not, by itself, stop Applebot from crawling the site and does not remove pages from Apple search and discovery experiences. That separation is the main reason Applebot and Applebot-Extended should not be treated as duplicate agents.

Search visibility and training use are separate choices

If you want Applebot to keep discovering public pages while opting out of the training use controlled by Applebot-Extended, keep the groups separate:

User-agent: Applebot
Allow: /

User-agent: Applebot-Extended
Disallow: /

If you instead block Applebot, you are changing crawl access for Apple search/discovery itself. Choose that only when crawl removal is actually your goal.

Do not use robots.txt as privacy protection

Applebot-Extended is a publisher preference signal. It is not authentication and it does not make a URL private. Pages that should not be public need normal application or server access controls.

Use the robots.txt Checker to verify the final groups and make sure a broad User-agent: * rule is not accidentally producing a different result than the Apple-specific rules.

Applebot-Extended and other Apple page signals

Apple documents other page-level controls for how crawled content can be used in product experiences, but they answer different questions. Do not substitute a snippet or presentation directive when your actual goal is the foundation-model training choice represented by Applebot-Extended. Conversely, do not block Applebot-Extended and assume that Applebot search crawling has stopped. Check each control against the outcome you want.

FAQ

Does Applebot-Extended crawl my website?

No. Apple says Applebot-Extended does not crawl webpages; Applebot performs the crawl and the Extended token controls how that data may be used.

Will blocking Applebot-Extended remove my pages from Apple search?

No. Apple states that pages disallowing Applebot-Extended can still appear in Apple search results.

What use does Applebot-Extended control?

It controls whether Applebot-crawled website content may be used to train Apple’s general-purpose foundation models for generative AI features.

Is nosnippet the same as Applebot-Extended?

No. Applebot-Extended addresses foundation-model training, while Apple documents nosnippet as a separate control for use of page content in broad world-knowledge answers.

Can I block only one directory from Applebot-Extended?

Yes. Use path-specific Disallow directives under the Applebot-Extended group instead of Disallow: / for the whole host.

Continue with the tools

Build a policy from the guidance above, then test the complete file that is actually published.

Use the generator Use the checker

Related pages

Public website content available to Apple search experiences while selected private paths remain blocked Applebot: robots.txt Rules for Apple Search and Discovery

Applebot is Apple’s web crawler for search and discovery experiences such as Spotlight, Siri and Safari. Blocking Applebot affects crawling; Applebot-Extended is a separate control for whether Applebot-fetched data may be used for training Apple’s generative foundation models.

 Website allowing search and AI discovery routes while blocking a separate model-training collection route Allow Search, Block AI Training robots.txt Template

Use separate user-agent groups: block documented training or dataset agents such as GPTBot and ClaudeBot, while allowing ordinary search and dedicated AI-search crawlers.

 Website selectively blocking AI training crawler routes while keeping search discovery routes open How to Block AI Crawlers with robots.txt

Block only the exact crawler names that match your policy, keep ordinary search and user-requested access separate when you still want them, publish the merged file at the correct host root, and verify each important crawler and path on the live site.

 Website blocking selected AI training and dataset routes while keeping search discovery open robots.txt for AI Training: Build a Selective Opt-Out

To restrict AI training access, identify each provider’s exact training or dataset token and disallow it in a dedicated robots.txt group. Keep search crawlers in separate allowed groups, treat product-control tokens such as Google-Extended according to their documentation, and use authentication or server-side controls when access must be enforced.

 Voluntary robots.txt crawler rules shown separately from enforceable server-side access controls Does robots.txt Stop AI Scraping? Limits and Better Controls

Robots.txt can stop compliant AI crawlers from requesting selected URLs, but it cannot enforce access against unidentified scrapers, spoofed user agents, browsers, or tools that ignore the protocol. Use precise crawler rules for policy, then add authentication, authorization, rate limits, or firewall controls where access must actually be prevented.