How to Allow PerplexityBot in robots.txt and Your Firewall
If your robots.txt disallows PerplexityBot, either by name or through a blanket `User-agent: *` rule, Perplexity cannot index your pages, so it rarely cites or links to them in its answers. This guide shows you how to find the rule that blocks it, add an explicit allow group, and make sure your CDN or firewall doesn't silently reject the crawler.

Why it matters
Perplexity builds its answers from pages that PerplexityBot has been allowed to crawl. Its cited sources appear as clickable links, so being crawlable is what makes your site eligible for that referral traffic. If the bot is blocked, Perplexity will usually describe your topic using competitors' pages or third-party summaries of your content.
Your brand loses the citation, the click and control over how your product is described. txt group plus one firewall exception, and it doesn't affect Googlebot or other crawlers.
What you need
Step-by-step fix
Fetch your live robots.txt
Start with the file crawlers actually receive, not the one in your CMS. A cache, plugin or CDN can serve something different from what you think you saved. txt in a browser, or fetch it from the command line.
Confirm it returns HTTP 200 with Content-Type: text/plain. A 5xx error or an HTML error page can cause crawlers to treat the whole site as off-limits.
curl -sI https://example.com/robots.txt curl -s https://example.com/robots.txt
Find every rule that blocks PerplexityBot
Robots.txt groups apply to the user agents listed directly above them. A crawler obeys the most specific group that matches its name and ignores the * group. Look for:
- A
User-agent: PerplexityBotgroup withDisallow: / - PerplexityBot stacked with other AI bots above a single
Disallow: / - No PerplexityBot group at all, combined with
User-agent: *andDisallow: / - Plugin-generated "block AI crawlers" blocks, often labelled with a comment
# These patterns all block PerplexityBot User-agent: PerplexityBot Disallow: / User-agent: GPTBot User-agent: PerplexityBot User-agent: ClaudeBot Disallow: / User-agent: * Disallow: /
Add an explicit PerplexityBot allow group
Remove PerplexityBot from any blocking group. Then add its own group with Allow: /. If you need to keep private paths out, such as /cart/ or /account/, add those Disallow lines inside the PerplexityBot group too.
Because the bot ignores the * group once it has its own, it won't inherit them from there. Keep the Sitemap line so the bot can find your URLs.
# robots.txt for example.com User-agent: PerplexityBot Allow: / Disallow: /cart/ Disallow: /account/ User-agent: OAI-SearchBot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: * Allow: / Disallow: /cart/ Disallow: /account/ Sitemap: https://example.com/sitemap.xml
Publish the change where robots.txt is generated
Save the change in the system that actually produces the file:
- A physical
robots.txtin your web root - Your SEO plugin's file editor
- A theme template
- Framework code
If both a physical file and a plugin exist, the physical file usually wins on WordPress, so edit that one. Then purge the robots.txt URL from your page cache and your CDN cache.
Allow PerplexityBot through your CDN or WAF
Robots.txt only grants permission. A firewall can still return 403 or a JavaScript challenge, which the crawler cannot solve. Check these places:
- AI-crawler blocking toggles
- Bot-fight modes
- Rate limits
- Custom rules that match "bot" or "crawler" in the user agent
Create a skip/allow rule for requests whose user agent contains PerplexityBot. Where your WAF supports it, pair that with Perplexity's published IP ranges (https://www.perplexity.com/perplexitybot.json) so spoofed user agents aren't let through.
# Cloudflare custom rule (Security → WAF → Custom rules), action: Skip
(http.user_agent contains "PerplexityBot" and ip.src in $perplexity_ips)
# Nginx: make sure no UA block catches it
if ($http_user_agent ~* "PerplexityBot") { set $allow_ai 1; }Test the response as PerplexityBot
Request both robots.txt and a normal content page using PerplexityBot's user agent string. Both should return 200 with real HTML content, not a 403, 429, a challenge page or an empty body. A pass here means at least your user-agent rules are fine.
UA="Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)"
curl -s -o /dev/null -w "%{http_code}\n" -A "$UA" https://example.com/robots.txt
curl -s -o /dev/null -w "%{http_code}\n" -A "$UA" https://example.com/blog/sample-post/Re-run the audit and watch your logs
Run the audit again to confirm PerplexityBot is now reported as allowed. Over the next days and weeks, filter your server or CDN logs for PerplexityBot to confirm real crawl hits are returning 200. How soon and how often it crawls varies by site.
grep -i "PerplexityBot" /var/log/nginx/access.log | awk '{print $9}' | sort | uniq -cPlatform-specific notes
WordPress
In Yoast SEO, go to Yoast SEO → Tools → File editor. In Rank Math, go to Rank Math → General Settings → Edit robots.txt. If a physical robots.txt exists in the web root, it overrides the plugin's virtual file, so edit it via SFTP instead. Some security plugins and "block AI bots" plugins add their own Disallow rules or server blocks, so check their settings too.
Shopify
Go to Online Store → Themes → … → Edit code and add a `robots.txt.liquid` template under Templates. Keep the default `{% for group in robots.default_groups %}` loop so Shopify's built-in rules stay. Then add a plain-text `User-agent: PerplexityBot` / `Allow: /` group after the loop.
Wix
Go to Marketing & SEO → SEO → Robots.txt Editor (under Tools and settings). Add the PerplexityBot group and save. Wix serves the file directly, and there's no separate firewall for you to configure.
Next.js
With the App Router, create `app/robots.ts` that exports a function returning `MetadataRoute.Robots`. For example: `rules: [{ userAgent: 'PerplexityBot', allow: '/' }, { userAgent: '*', allow: '/' }]` plus `sitemap: 'https://example.com/sitemap.xml'`. Delete any static `public/robots.txt`, because it conflicts with the generated file. Also check middleware and hosting firewalls, such as Vercel Firewall rules, for user-agent blocks.
Cloudflare
Check Security → Bots and the AI Crawl Control section (formerly AI Audit). Cloudflare can block AI crawlers by default on newer zones, and it can also inject Disallow rules into a managed robots.txt. Make sure PerplexityBot is set to Allow there. Then add a WAF custom rule with the Skip action for its user agent combined with Perplexity's IP list.
Check that it worked
✅ How to verify
- Open https://example.com/robots.txt and confirm a `User-agent: PerplexityBot` group with `Allow: /` is present and no group that applies to it says `Disallow: /`.
- Run curl with the PerplexityBot user agent against robots.txt and a content page; both should return HTTP 200 with real content.
- Check your CDN/WAF event log for requests from PerplexityBot and confirm none are blocked or challenged.
- Re-run the audit; the PerplexityBot access check should pass.
⚠️ Common mistakes
- Adding `Allow: /` under `User-agent: *` while a separate PerplexityBot group still says `Disallow: /`. The named group wins.
- Fixing robots.txt but leaving a Cloudflare AI-bot block, Bot Fight Mode or a WAF rule that returns 403 or a challenge to the crawler.
- Editing a plugin's virtual robots.txt while a physical robots.txt file in the web root overrides it, or forgetting to purge the cached file.
- Allowing any request whose user agent contains "PerplexityBot" without IP verification, which lets scrapers spoof it to bypass your protections.
Is this still an issue on your site? Run a free audit — ZORO checks this and 294 other SEO, AI-visibility, accessibility and security signals, with fixes for your platform.
Check my site freeFAQ
What is the difference between PerplexityBot and Perplexity-User?
PerplexityBot is the crawler that indexes pages so Perplexity can surface and cite them in answers, and it respects robots.txt. Perplexity-User fetches a page when a person asks Perplexity about a specific URL. Perplexity says it generally doesn't apply robots.txt to these user-initiated fetches. This audit check is about PerplexityBot.
Does allowing PerplexityBot let Perplexity train AI models on my content?
Perplexity describes PerplexityBot as a crawler that surfaces and links websites in its search results, not as a crawler for training foundation models. Review Perplexity's current documentation and terms if training use matters to you. Robots.txt controls crawling only.
How long until my site shows up in Perplexity answers after unblocking?
Crawling can start within days once the bot can reach you, but there's no fixed schedule. Appearing as a cited source also depends on whether your pages answer the question well and are easy to parse. A sitemap and fast, server-rendered HTML help.
Can I allow PerplexityBot on some sections but not others?
Yes. Inside the `User-agent: PerplexityBot` group, use `Allow: /` plus `Disallow:` lines for the paths you want excluded, such as `/members/` or `/checkout/`. Remember it won't inherit Disallow rules from the `*` group once it has its own group.
How to Set an Explicit Google-Extended Policy in robots.txt
How to Allow ClaudeBot and Claude-SearchBot in robots.txt
How to Fix a CDN or WAF Blocking AI Crawlers
How to Allow GPTBot and OAI-SearchBot in robots.txt
How to Create and Publish an llms.txt File for AI Tools
How to Fix AI Overview Eligibility by Ranking in the Top 10