Applebot robots.txt Rules for Siri, Spotlight, and Safari

What Applebot does

Applebot crawls public web pages for search and discovery features across Apple products, including Siri, Spotlight, and Safari. Allowing it can help eligible pages appear when Apple users search or request current web information. Blocking Applebot affects this Apple discovery path, but it does not directly block Googlebot, Bingbot, or other crawlers with separate robots.txt groups.

How Applebot reads robots.txt

Applebot respects standard robots.txt directives addressed to Applebot. A specific Applebot group is the clearest way to express your policy. Apple also documents an important fallback: when robots.txt does not mention Applebot but does contain rules for Googlebot, Applebot follows the Googlebot instructions. That means an old Googlebot restriction can influence Applebot even when no Applebot token appears in the file.

Applebot does not support the non-standard Crawl-delay directive. If request volume becomes a concern, avoid relying on Crawl-delay for this agent. Apple says its crawler adjusts automatically when a server slows down or returns errors.

Allow public pages and exclude private sections

The example below keeps general Apple search discovery open while excluding two non-public directories. Replace the paths with real sections from your site. For confidential content, robots.txt is not enough: use authentication, authorization, and server-side access controls because the file is only an instruction to compliant crawlers.

Publish robots.txt at the root of each host, such as https://example.com/robots.txt. A rule on the main domain does not automatically apply to docs.example.com or another subdomain.

Rendering and indexing controls

Applebot may render pages in a browser. Blocking required JavaScript, CSS, images, or XHR endpoints can prevent correct rendering even when the HTML page itself is allowed. Use the robots.txt checker to catch broad resource restrictions before publishing them.

Robots.txt controls crawling, not every indexing or presentation decision. Applebot also supports page-level robots meta directives such as noindex. Apple separately documents nosnippet and paywall structured data for specific content-use and display scenarios.

Applebot versus Applebot-Extended

Do not treat these tokens as interchangeable. Applebot performs the actual search crawl. Applebot-Extended is a separate control for whether data crawled by Applebot may be used to train Apple’s general-purpose foundation models. You can allow Applebot for search while disallowing Applebot-Extended for training use.

Verification checklist

1. Place robots.txt at the host root.
2. Add an explicit Applebot group instead of relying unintentionally on Googlebot fallback.
3. Keep render-critical assets accessible.
4. Check each subdomain separately.
5. Run the published file through the checker.