PerplexityBot robots.txt Rules for Search Visibility
PerplexityBot is Perplexity’s automated search crawler, not a foundation-model training crawler. Allow it if you want public pages discovered and cited in Perplexity; block it with a specific robots.txt group if you do not.
Primary reference
Official crawler documentation
Use the provider's current documentation to confirm the crawler token and purpose before changing a production robots.txt file.
Copy-paste robots.txt example
User-agent: PerplexityBot Allow: / Sitemap: https://example.com/sitemap.xml
What PerplexityBot does
PerplexityBot is Perplexity’s automated crawler for discovering, indexing, and surfacing websites in Perplexity search results. Perplexity’s documentation says this agent is not used to crawl content for training AI foundation models. Its purpose is search discovery and source linking.
That distinction matters for publishers who want referral visibility from answer engines but do not want to make one broad decision for every AI-related agent. PerplexityBot and Perplexity-User have separate roles and should be evaluated independently.
When to allow PerplexityBot
Allow PerplexityBot when your goal is to make public articles, product information, documentation, or reference pages discoverable and linkable in Perplexity answers. Perplexity recommends allowing the bot and permitting requests from its published IP ranges if a firewall would otherwise block them.
The simplest allow rule is shown below. If your site already allows compliant crawlers by default, an explicit group is optional, but adding one can make your policy easier to audit and maintain.
How to block or limit it
To exclude the whole host, use Disallow: / under User-agent: PerplexityBot. To protect selected areas, list only those paths. Do not use a broad wildcard block unless you intend to affect ordinary search crawlers as well.
Perplexity notes that settings work independently and may take up to 24 hours to be reflected. After changing the file, verify the live response at the domain root and test it with the robots.txt checker.
Verify real requests
A user-agent string can be spoofed. For firewall rules or log analysis, compare the request source with Perplexity’s current published IP lists rather than trusting the header alone. Keep those ranges dynamic because they may change.
PerplexityBot versus Perplexity-User
PerplexityBot performs automated search crawling. Perplexity-User fetches a page after a person asks Perplexity a question and, according to Perplexity, generally ignores robots.txt because the request is user initiated. See the dedicated comparison guide before applying the same rule to both.
For a ready-made starting point, use the Perplexity crawler template or create a tailored file in the generator.
FAQ
Is PerplexityBot used to train foundation models?
Perplexity states that PerplexityBot is designed for search discovery and linking and is not used to crawl content for AI foundation-model training.
Should I allow PerplexityBot?
Allow it if you want eligible public pages discovered and linked in Perplexity search results. Block or limit it if that visibility is not part of your publishing policy.
How long can a robots.txt change take to affect Perplexity?
Perplexity says changes may take up to 24 hours to be reflected by its systems.
How can I verify that a request is really from Perplexity?
Check both the user-agent and the current official Perplexity IP ranges. A user-agent header alone can be spoofed.
Is PerplexityBot the same as Perplexity-User?
No. PerplexityBot is an automated search crawler, while Perplexity-User performs user-requested retrieval and generally ignores robots.txt.
Perplexity-User robots.txt Limits and Access Control
Block Perplexity Crawlers robots.txt Template
PerplexityBot vs Perplexity-User: What Changes?
robots.txt for AI Search Visibility and Citations
robots.txt Checker