PerplexityBot automatically discovers pages for Perplexity search and is governed by its own robots.txt token. Perplexity-User fetches a page after a person asks a question; Perplexity says this user-triggered agent generally ignores robots.txt. Use precise crawler rules for search visibility, and use server-side controls when access must be enforced.
User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: /
Perplexity operates two published user agents that can reach a website for different reasons. PerplexityBot performs automatic discovery for Perplexity search. Perplexity-User makes a request in response to an individual user action. Treating them as interchangeable leads to incorrect robots.txt policies and misleading expectations about enforcement.
PerplexityBot is designed to find public pages that may be surfaced and linked in Perplexity search results. Perplexity states that this bot is not used to crawl content for training AI foundation models. If visibility and citations in Perplexity matter to your site, allow this token and ensure your firewall does not block its verified IP ranges.
Blocking PerplexityBot is therefore a search-discovery decision. It can reduce Perplexity’s ability to discover or refresh your pages, but it does not secure the content from browsers, unidentified scrapers, or user-triggered fetchers.
Perplexity-User may visit one page when a person asks Perplexity a question that requires current web content. It is not an automatic indexing crawler and is not used to collect material for foundation-model training. Perplexity’s current documentation says that, because the fetch is user initiated, this agent generally ignores robots.txt rules.
That distinction matters. A Disallow group for Perplexity-User can express a preference, but it should not be treated as reliable access enforcement. If a page must not be fetched, protect it with authentication, authorization, a paywall, or a carefully designed WAF rule.
For most public editorial and commercial sites, the simplest visibility-first policy is to allow PerplexityBot. You may also allow Perplexity-User, while recognizing that the user agent’s behavior is not equivalent to automatic crawler compliance.
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /If you want to opt out of automatic Perplexity search discovery but still permit user-requested access, block only the search bot:
User-agent: PerplexityBot
Disallow: /
User-agent: Perplexity-User
Allow: /User-agent strings can be spoofed. When configuring a WAF or analyzing logs, compare the source address with Perplexity’s published IP files rather than trusting the header by itself. A WAF can also accidentally block legitimate traffic even when robots.txt says Allow: /, so check both layers.
For broader policy design, see robots.txt for AI Search Visibility and How to Block AI Crawlers.
Perplexity states that PerplexityBot is used to surface and link websites in search results and is not used to crawl content for AI foundation-model training.
Perplexity says the fetch is initiated by a user and therefore Perplexity-User generally ignores robots.txt. Use server-side access controls when a restriction must be enforced.
You can publish separate groups, but a Disallow rule for Perplexity-User should be viewed as a preference rather than dependable enforcement because the provider says user-triggered fetches generally ignore robots.txt.
It can prevent automatic discovery and reduce search visibility, but previously known URLs, navigational links, or user-requested access may behave differently. The two agents are controlled independently.
Check both the user-agent string and the request source against Perplexity’s officially published IP address files. A user-agent header alone can be spoofed.
PerplexityBot robots.txt Rules for Search VisibilityPerplexityBot is Perplexity’s automated search crawler, not a foundation-model training crawler. Allow it if you want public pages discovered and cited in Perplexity; block it with a specific robots.txt group if you do not.
Perplexity-User robots.txt Limits and Access ControlPerplexity-User retrieves pages after a person asks Perplexity a question. Perplexity says these user-triggered fetches generally ignore robots.txt, so private content must be protected with authentication, authorization, or WAF rules.
Robots.txt Checker for AI CrawlersEnter a domain, choose a crawler, and test a path. The checker downloads the live robots.txt file, finds the applicable User-agent group and most specific rule, then explains whether access is allowed or blocked.
How to Block AI Crawlers with robots.txtTo block an AI crawler, add a group for its exact User-agent token and use Disallow: /. Block only the crawler purposes you intend to restrict, keep ordinary search crawlers open when visibility matters, and remember that robots.txt is a request to compliant bots rather than a security control.
robots.txt for AI Search Visibility and CitationsTo support AI search visibility, allow each provider’s documented search crawler—such as OAI-SearchBot, Claude-SearchBot, and PerplexityBot—while keeping Googlebot and Bingbot accessible. Robots.txt only permits crawling; it does not guarantee indexing, citation, ranking, or access through a firewall.