PerplexityBot vs Perplexity-User

PerplexityBot is the automatic crawler used to surface and link sites in Perplexity search results. Perplexity-User supports user-requested fetches; Perplexity states that this fetcher generally ignores robots.txt because the request is user initiated.

Independent technical reference Updated Editorial approach Report a correction
Automatic Perplexity search crawler compared with a single user-triggered page request

Copy-paste robots.txt example

User-agent: PerplexityBot
Allow: /

# Perplexity-User is user-triggered; Perplexity states it generally ignores robots.txt.
User-agent: Perplexity-User
Allow: /

Perplexity publishes two user agents with very different behavior. Treating them as interchangeable can produce a robots.txt policy that does not match your intent.

Perplexity example: separate discovery from a user's fetch

Suppose a publisher permits the automatic search crawler to discover public reporting but requires a login for subscriber-only reports. The right test has two components: a robots.txt rule for PerplexityBot and a real authorization check on the private content. A Perplexity-User request should never be the only layer preventing an unauthorized read.

User-agent: PerplexityBot
Disallow: /subscriber/
Allow: /

Test /news/briefing (intended allowed) and /subscriber/report (intended blocked for PerplexityBot). Then check the subscriber URL as a signed-out visitor: it should not reveal confidential report text. The second test is mandatory because user-directed retrieval may not behave like automatic crawler traffic.

Review current guidance at Perplexity crawler documentation and compare with post-deployment verification. Allowing an automated crawler only makes content eligible for discovery; it does not guarantee that Perplexity will index, quote or send visitors to it.

PerplexityBot: automatic search crawling

PerplexityBot is designed to discover and surface websites in Perplexity search results. Perplexity says it is not used to crawl content for AI foundation-model training and recommends allowing it when you want the site to appear in search results.

User-agent: PerplexityBot
Allow: /

Perplexity-User: user-triggered retrieval

Perplexity-User can fetch a page when a user asks Perplexity a question. Perplexity says this agent is not used for automatic web crawling or foundation-model training. Its current documentation also says that, because the fetch is user requested, Perplexity-User generally ignores robots.txt rules.

That means a Disallow entry should not be presented as reliable access control for Perplexity-User. If the page must not be retrieved publicly, enforce the restriction at the application, server or WAF layer.

Choose based on the actual goal

  • Want Perplexity search discovery? Allow PerplexityBot and ensure your WAF does not accidentally block its verified traffic.
  • Want to stop automatic Perplexity search crawling? Disallow PerplexityBot.
  • Need to protect private content? Do not rely on a Perplexity-User robots.txt rule; require real access control.

Verify traffic, not just the User-Agent string

User-Agent strings can be spoofed. Perplexity publishes IP ranges for its agents and recommends combining User-Agent and IP verification in WAF configurations. For the bot reference, see PerplexityBot. To test robots behavior, use the robots.txt Checker.

That distinction is especially important during incident response: a robots.txt test can validate PerplexityBot policy, but it cannot prove that a user-triggered Perplexity-User fetch will be denied.

FAQ

Is PerplexityBot used to train foundation models?

Perplexity states that PerplexityBot is used to surface and link websites in search results and is not used to crawl content for AI foundation-model training.

Does Perplexity-User obey robots.txt?

Perplexity says the fetch is initiated by a user and therefore Perplexity-User generally ignores robots.txt. Use server-side access controls when a restriction must be enforced.

Can I allow PerplexityBot but block Perplexity-User?

You can publish separate groups, but a Disallow rule for Perplexity-User should be viewed as a preference rather than dependable enforcement because the provider says user-triggered fetches generally ignore robots.txt.

Will blocking PerplexityBot remove my site from every Perplexity answer?

It can prevent automatic discovery and reduce search visibility, but previously known URLs, navigational links, or user-requested access may behave differently. The two agents are controlled independently.

How can I verify a real Perplexity crawler?

Check both the user-agent string and the request source against Perplexity’s officially published IP address files. A user-agent header alone can be spoofed.

Continue with the tools

Build a policy from the guidance above, then test the complete file that is actually published.

Use the generator Use the checker

Related pages

Perplexity search crawler discovering public websites and carrying selected sources into a cited answer PerplexityBot robots.txt Rules for Search Visibility

PerplexityBot is Perplexity’s automated search crawler, not a foundation-model training crawler. Allow it if you want public pages discovered and cited in Perplexity; block it with a specific robots.txt group if you do not.

 User-triggered web request passing an advisory robots.txt signal before reaching real server-side authentication Perplexity-User robots.txt Limits and Access Control

Perplexity-User retrieves pages after a person asks Perplexity a question. Perplexity says these user-triggered fetches generally ignore robots.txt, so private content must be protected with authentication, authorization, or WAF rules.

 Robots.txt checker inspecting a website and reporting allowed and blocked crawler access robots.txt Checker

Test the live robots.txt served by a website or paste unpublished rules to simulate a crawler across up to eight URL paths. Review the winning Allow or Disallow rule before deploying changes.

 Website selectively blocking AI training crawler routes while keeping search discovery routes open How to Block AI Crawlers with robots.txt

Block only the exact crawler names that match your policy, keep ordinary search and user-requested access separate when you still want them, publish the merged file at the correct host root, and verify each important crawler and path on the live site.

 Public pages moving through AI search discovery and technical access toward possible source citation robots.txt for AI Search: Discovery Crawler Guide

If AI-search visibility matters, identify the agents used for search/discovery separately from training crawlers. OAI-SearchBot, Claude-SearchBot, PerplexityBot and Applebot have discovery roles, while similarly named training or user-triggered agents may need different policies.