Use a Disallow group for PerplexityBot to opt out of Perplexity’s automated search indexing. You may also publish a Perplexity-User rule, but Perplexity states that this user-triggered fetcher generally ignores robots.txt, so enforce a real block at the server or application layer.
User-agent: PerplexityBot Disallow: / User-agent: Perplexity-User Disallow: /
Perplexity documents two agents with different behavior. PerplexityBot automatically discovers and indexes public pages so they can be surfaced and linked in Perplexity search results. Perplexity-User fetches a page after a person asks Perplexity a question. The two settings are independent.
This template expresses a block for both tokens:
User-agent: PerplexityBot
Disallow: /
User-agent: Perplexity-User
Disallow: /The first group is the reliable robots.txt control for Perplexity’s automated crawler. The second records your preference for user-triggered retrieval, but Perplexity’s current documentation says that Perplexity-User generally ignores robots.txt because the fetch was requested by a user.
Neither group blocks Googlebot, Bingbot, or another crawler unless your file also contains broader rules that apply to them.
When you need to prevent Perplexity-User from reaching a resource, evaluate the request at the network or application layer. Perplexity publishes separate IP information for its agents, which can support verification, but do not trust the User-Agent header alone because headers can be spoofed. Suitable controls may include authentication, authorization, rate limits, a web application firewall, or a carefully maintained allow/deny policy.
Do not use robots.txt to protect private data, paid content, checkout pages, or administration areas. The file is public and compliance is voluntary.
https://your-domain.com/robots.txt as plain text.PerplexityBot and Perplexity-User separately with the robots.txt checker.If you want Perplexity search visibility but not unrestricted user-triggered access, allow PerplexityBot and apply enforceable controls only to the resources that should not be retrieved on demand.
Perplexity documents PerplexityBot as the automated crawler used to surface and link websites in search results, and provides robots.txt controls for it.
Not reliably. Perplexity states that Perplexity-User generally ignores robots.txt because the page fetch is initiated by a user request.
No. A PerplexityBot-specific group does not apply to Googlebot. It can, however, reduce how your site is discovered and cited within Perplexity.
Use server-side controls such as authentication, authorization, verified network rules, rate limits, or a web application firewall. Do not rely on the User-Agent string alone.
Yes. Replace the site-wide slash with specific path rules, then test both the blocked and allowed URLs before publishing the final file.
PerplexityBot robots.txt Rules for Search VisibilityPerplexityBot is Perplexity’s automated search crawler, not a foundation-model training crawler. Allow it if you want public pages discovered and cited in Perplexity; block it with a specific robots.txt group if you do not.
Perplexity-User robots.txt Limits and Access ControlPerplexity-User retrieves pages after a person asks Perplexity a question. Perplexity says these user-triggered fetches generally ignore robots.txt, so private content must be protected with authentication, authorization, or WAF rules.
PerplexityBot vs Perplexity-User: What Changes?PerplexityBot automatically discovers pages for Perplexity search and is governed by its own robots.txt token. Perplexity-User fetches a page after a person asks a question; Perplexity says this user-triggered agent generally ignores robots.txt. Use precise crawler rules for search visibility, and use server-side controls when access must be enforced.
Robots.txt Checker for AI CrawlersEnter a domain, choose a crawler, and test a path. The checker downloads the live robots.txt file, finds the applicable User-agent group and most specific rule, then explains whether access is allowed or blocked.
How to Block AI Crawlers with robots.txtTo block an AI crawler, add a group for its exact User-agent token and use Disallow: /. Block only the crawler purposes you intend to restrict, keep ordinary search crawlers open when visibility matters, and remember that robots.txt is a request to compliant bots rather than a security control.