PerplexityBot robots.txt Rules for Search Visibility

What PerplexityBot does

PerplexityBot is Perplexity’s automated crawler for discovering, indexing, and surfacing websites in Perplexity search results. Perplexity’s documentation says this agent is not used to crawl content for training AI foundation models. Its purpose is search discovery and source linking.

Perplexity search visibility: test the actual outcome

If your goal is to let Perplexity discover and surface a public article, test PerplexityBot on the article URL rather than checking an unrelated bot. For example, a restrictive wildcard group and a dedicated search group behave differently:

User-agent: *
Disallow: /members/

User-agent: PerplexityBot
Disallow: /members/
Allow: /

In this example, /news/story should be allowed to PerplexityBot and /members/report should be blocked. The specific PerplexityBot group is evaluated on its own; it does not automatically inherit wildcard restrictions. Check the allowed and blocked paths individually in the draft simulator, then retest the published file.

The second Perplexity identity, Perplexity-User, is used in a different user-directed context. Do not assume a rule for automatic PerplexityBot crawling enforces access control on all user-initiated retrieval. Search eligibility is also not a guarantee of inclusion, citation or traffic. Keep genuinely private documents behind authentication.

That distinction matters for publishers who want referral visibility from answer engines but do not want to make one broad decision for every AI-related agent. PerplexityBot and Perplexity-User have separate roles and should be evaluated independently.

When to allow PerplexityBot

Allow PerplexityBot when your goal is to make public articles, product information, documentation, or reference pages discoverable and linkable in Perplexity answers. Perplexity recommends allowing the bot and permitting requests from its published IP ranges if a firewall would otherwise block them.

The simplest allow rule is shown below. If your site already allows compliant crawlers by default, an explicit group is optional, but adding one can make your policy easier to audit and maintain.

How to block or limit it

To exclude the whole host, use Disallow: / under User-agent: PerplexityBot. To protect selected areas, list only those paths. Do not use a broad wildcard block unless you intend to affect ordinary search crawlers as well.

Perplexity notes that settings work independently and may take up to 24 hours to be reflected. After changing the file, verify the live response at the domain root and test it with the robots.txt checker.

Verify real requests

A user-agent string can be spoofed. For firewall rules or log analysis, compare the request source with Perplexity’s current published IP lists rather than trusting the header alone. Keep those ranges dynamic because they may change.

PerplexityBot versus Perplexity-User

PerplexityBot performs automated search crawling. Perplexity-User fetches a page after a person asks Perplexity a question and, according to Perplexity, generally ignores robots.txt because the request is user initiated. See the dedicated comparison guide before applying the same rule to both.

For a ready-made starting point, use the Perplexity crawler template or create a tailored file in the generator.