Perplexity-User retrieves pages after a person asks Perplexity a question. Perplexity says these user-triggered fetches generally ignore robots.txt, so private content must be protected with authentication, authorization, or WAF rules.
# Advisory signal only: Perplexity says user-requested fetches generally ignore robots.txt. # Protect private content with authentication, authorization, or WAF rules. User-agent: Perplexity-User Disallow: /
Perplexity-User is a user-triggered fetcher. When a person asks Perplexity a question, the service may request a webpage to produce a more accurate answer and include a link to the source. Perplexity states that this agent is not used for automated web crawling or for collecting content to train AI foundation models.
Perplexity’s current documentation says Perplexity-User generally ignores robots.txt because the fetch was requested by a user. That makes this page fundamentally different from a normal crawler guide: a Disallow rule may express your preference, but it should not be treated as an enforceable block.
The example below is therefore advisory only. Do not use it as proof that private or paid content is protected. A public robots.txt file cannot replace application security.
Perplexity publishes separate current IP lists for PerplexityBot and Perplexity-User. For WAF decisions, validate both the user-agent and source address because headers can be spoofed.
PerplexityBot is the automated search crawler and follows robots.txt. Perplexity-User is a direct, user-initiated fetch. Blocking PerplexityBot may reduce search discovery, while controlling Perplexity-User requires actual access enforcement when the content is not public.
Keep genuinely public pages accessible if you want them cited. Secure genuinely private pages at the server. Treat the robots.txt token as a policy signal only, then verify your application behavior with an unauthenticated request. The robots.txt security guide explains the broader limitation, and the comparison page covers both Perplexity agents.
Perplexity says user-triggered Perplexity-User fetches generally ignore robots.txt, so a Disallow rule should not be treated as reliable access control.
Perplexity states that Perplexity-User is not used for web crawling or collecting content to train AI foundation models.
Protect those pages with authentication, authorization, signed URLs, server rules, or a WAF. Do not rely on robots.txt alone.
Perplexity publishes a current IP list for Perplexity-User. For security decisions, check both the source address and user-agent because either signal alone may be insufficient.
PerplexityBot performs automated search discovery and follows robots.txt. Perplexity-User fetches a page because a user requested it and generally ignores robots.txt.
PerplexityBot robots.txt Rules for Search VisibilityPerplexityBot is Perplexity’s automated search crawler, not a foundation-model training crawler. Allow it if you want public pages discovered and cited in Perplexity; block it with a specific robots.txt group if you do not.
Block Perplexity Crawlers robots.txt TemplateUse a Disallow group for PerplexityBot to opt out of Perplexity’s automated search indexing. You may also publish a Perplexity-User rule, but Perplexity states that this user-triggered fetcher generally ignores robots.txt, so enforce a real block at the server or application layer.
PerplexityBot vs Perplexity-User: What Changes?PerplexityBot automatically discovers pages for Perplexity search and is governed by its own robots.txt token. Perplexity-User fetches a page after a person asks a question; Perplexity says this user-triggered agent generally ignores robots.txt. Use precise crawler rules for search visibility, and use server-side controls when access must be enforced.
Does robots.txt Stop AI Scraping? Limits and Better ControlsRobots.txt can stop compliant AI crawlers from requesting selected URLs, but it cannot enforce access against unidentified scrapers, spoofed user agents, browsers, or tools that ignore the protocol. Use precise crawler rules for policy, then add authentication, authorization, rate limits, or firewall controls where access must actually be prevented.
Robots.txt Checker for AI CrawlersEnter a domain, choose a crawler, and test a path. The checker downloads the live robots.txt file, finds the applicable User-agent group and most specific rule, then explains whether access is allowed or blocked.
