PerplexityBot vs Perplexity-User: What Site Owners Control

PerplexityBot automatically discovers pages for Perplexity search and is governed by its own robots.txt token. Perplexity-User fetches a page after a person asks a question; Perplexity says this user-triggered agent generally ignores robots.txt. Use precise crawler rules for search visibility, and use server-side controls when access must be enforced.

Independent technical reference Updated Editorial approach Report a correction
Automatic Perplexity search crawler compared with a single user-triggered page request

Copy-paste robots.txt example

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

Two agents with different triggers

Perplexity operates two published user agents that can reach a website for different reasons. PerplexityBot performs automatic discovery for Perplexity search. Perplexity-User makes a request in response to an individual user action. Treating them as interchangeable leads to incorrect robots.txt policies and misleading expectations about enforcement.

PerplexityBot supports search discovery

PerplexityBot is designed to find public pages that may be surfaced and linked in Perplexity search results. Perplexity states that this bot is not used to crawl content for training AI foundation models. If visibility and citations in Perplexity matter to your site, allow this token and ensure your firewall does not block its verified IP ranges.

Blocking PerplexityBot is therefore a search-discovery decision. It can reduce Perplexity’s ability to discover or refresh your pages, but it does not secure the content from browsers, unidentified scrapers, or user-triggered fetchers.

Perplexity-User follows a person’s request

Perplexity-User may visit one page when a person asks Perplexity a question that requires current web content. It is not an automatic indexing crawler and is not used to collect material for foundation-model training. Perplexity’s current documentation says that, because the fetch is user initiated, this agent generally ignores robots.txt rules.

That distinction matters. A Disallow group for Perplexity-User can express a preference, but it should not be treated as reliable access enforcement. If a page must not be fetched, protect it with authentication, authorization, a paywall, or a carefully designed WAF rule.

Choose the policy that matches your goal

For most public editorial and commercial sites, the simplest visibility-first policy is to allow PerplexityBot. You may also allow Perplexity-User, while recognizing that the user agent’s behavior is not equivalent to automatic crawler compliance.

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

If you want to opt out of automatic Perplexity search discovery but still permit user-requested access, block only the search bot:

User-agent: PerplexityBot
Disallow: /

User-agent: Perplexity-User
Allow: /

Do not rely on the name alone

User-agent strings can be spoofed. When configuring a WAF or analyzing logs, compare the source address with Perplexity’s published IP files rather than trusting the header by itself. A WAF can also accidentally block legitimate traffic even when robots.txt says Allow: /, so check both layers.

Verify the result

  1. Publish robots.txt at the root of the exact host.
  2. Run it through the robots.txt checker.
  3. Confirm that no wildcard group accidentally blocks the agents you intended to allow.
  4. Review access logs after the provider’s stated update window.
  5. Use server-side protection for content that must remain private.

For broader policy design, see robots.txt for AI Search Visibility and How to Block AI Crawlers.

FAQ

Is PerplexityBot used to train foundation models?

Perplexity states that PerplexityBot is used to surface and link websites in search results and is not used to crawl content for AI foundation-model training.

Does Perplexity-User obey robots.txt?

Perplexity says the fetch is initiated by a user and therefore Perplexity-User generally ignores robots.txt. Use server-side access controls when a restriction must be enforced.

Can I allow PerplexityBot but block Perplexity-User?

You can publish separate groups, but a Disallow rule for Perplexity-User should be viewed as a preference rather than dependable enforcement because the provider says user-triggered fetches generally ignore robots.txt.

Will blocking PerplexityBot remove my site from every Perplexity answer?

It can prevent automatic discovery and reduce search visibility, but previously known URLs, navigational links, or user-requested access may behave differently. The two agents are controlled independently.

How can I verify a real Perplexity crawler?

Check both the user-agent string and the request source against Perplexity’s officially published IP address files. A user-agent header alone can be spoofed.

Continue with the tools

Build a policy from the guidance above, then test the complete file that is actually published.

Use the generator Use the checker

Related pages

Perplexity search crawler discovering public websites and carrying selected sources into a cited answer PerplexityBot robots.txt Rules for Search Visibility

PerplexityBot is Perplexity’s automated search crawler, not a foundation-model training crawler. Allow it if you want public pages discovered and cited in Perplexity; block it with a specific robots.txt group if you do not.

 User-triggered web request passing an advisory robots.txt signal before reaching real server-side authentication Perplexity-User robots.txt Limits and Access Control

Perplexity-User retrieves pages after a person asks Perplexity a question. Perplexity says these user-triggered fetches generally ignore robots.txt, so private content must be protected with authentication, authorization, or WAF rules.

 Robots.txt checker inspecting a website and reporting allowed and blocked crawler access robots.txt Checker

Enter the exact website address, choose a crawler, and test a path. The checker reads the live robots.txt file and explains the group and rule that determine access.

 Website selectively blocking AI training crawler routes while keeping search discovery routes open How to Block AI Crawlers with robots.txt

Use the exact published User-agent name for each crawler you intend to restrict, add Disallow: / to that group, preserve any search or user-requested access you still want, then publish and test the live file. Do not use User-agent: * unless you intend to restrict ordinary search crawlers too.

 Public pages moving through AI search discovery and technical access toward possible source citation robots.txt for AI Search Visibility and Citations

To support AI search visibility, allow each provider’s documented search crawler—such as OAI-SearchBot, Claude-SearchBot, and PerplexityBot—while keeping Googlebot and Bingbot accessible. Robots.txt only permits crawling; it does not guarantee indexing, citation, ranking, or access through a firewall.