PerplexityBot is Perplexity’s automated search crawler, not a foundation-model training crawler. Allow it if you want public pages discovered and cited in Perplexity; block it with a specific robots.txt group if you do not.
User-agent: PerplexityBot Allow: / Sitemap: https://example.com/sitemap.xml
PerplexityBot is Perplexity’s automated crawler for discovering, indexing, and surfacing websites in Perplexity search results. Perplexity’s documentation says this agent is not used to crawl content for training AI foundation models. Its purpose is search discovery and source linking.
That distinction matters for publishers who want referral visibility from answer engines but do not want to make one broad decision for every AI-related agent. PerplexityBot and Perplexity-User have separate roles and should be evaluated independently.
Allow PerplexityBot when your goal is to make public articles, product information, documentation, or reference pages discoverable and linkable in Perplexity answers. Perplexity recommends allowing the bot and permitting requests from its published IP ranges if a firewall would otherwise block them.
The simplest allow rule is shown below. If your site already allows compliant crawlers by default, an explicit group is optional, but adding one can make your policy easier to audit and maintain.
To exclude the whole host, use Disallow: / under User-agent: PerplexityBot. To protect selected areas, list only those paths. Do not use a broad wildcard block unless you intend to affect ordinary search crawlers as well.
Perplexity notes that settings work independently and may take up to 24 hours to be reflected. After changing the file, verify the live response at the domain root and test it with the robots.txt checker.
A user-agent string can be spoofed. For firewall rules or log analysis, compare the request source with Perplexity’s current published IP lists rather than trusting the header alone. Keep those ranges dynamic because they may change.
PerplexityBot performs automated search crawling. Perplexity-User fetches a page after a person asks Perplexity a question and, according to Perplexity, generally ignores robots.txt because the request is user initiated. See the dedicated comparison guide before applying the same rule to both.
For a ready-made starting point, use the Perplexity crawler template or create a tailored file in the generator.
Perplexity states that PerplexityBot is designed for search discovery and linking and is not used to crawl content for AI foundation-model training.
Allow it if you want eligible public pages discovered and linked in Perplexity search results. Block or limit it if that visibility is not part of your publishing policy.
Perplexity says changes may take up to 24 hours to be reflected by its systems.
Check both the user-agent and the current official Perplexity IP ranges. A user-agent header alone can be spoofed.
No. PerplexityBot is an automated search crawler, while Perplexity-User performs user-requested retrieval and generally ignores robots.txt.
Perplexity-User robots.txt Limits and Access ControlPerplexity-User retrieves pages after a person asks Perplexity a question. Perplexity says these user-triggered fetches generally ignore robots.txt, so private content must be protected with authentication, authorization, or WAF rules.
Block Perplexity Crawlers robots.txt TemplateUse a Disallow group for PerplexityBot to opt out of Perplexity’s automated search indexing. You may also publish a Perplexity-User rule, but Perplexity states that this user-triggered fetcher generally ignores robots.txt, so enforce a real block at the server or application layer.
PerplexityBot vs Perplexity-User: What Changes?PerplexityBot automatically discovers pages for Perplexity search and is governed by its own robots.txt token. Perplexity-User fetches a page after a person asks a question; Perplexity says this user-triggered agent generally ignores robots.txt. Use precise crawler rules for search visibility, and use server-side controls when access must be enforced.
robots.txt for AI Search Visibility and CitationsTo support AI search visibility, allow each provider’s documented search crawler—such as OAI-SearchBot, Claude-SearchBot, and PerplexityBot—while keeping Googlebot and Bingbot accessible. Robots.txt only permits crawling; it does not guarantee indexing, citation, ranking, or access through a firewall.
Robots.txt Checker for AI CrawlersEnter a domain, choose a crawler, and test a path. The checker downloads the live robots.txt file, finds the applicable User-agent group and most specific rule, then explains whether access is allowed or blocked.
