How to Allow PerplexityBot in robots.txt and Your Firewall

If your robots.txt disallows PerplexityBot, either by name or through a blanket `User-agent: *` rule, Perplexity cannot index your pages, so it rarely cites or links to them in its answers. This guide shows you how to find the rule that blocks it, add an explicit allow group, and make sure your CDN or firewall doesn't silently reject the crawler.

By Updated ⏱ 15 minutes🎯 Beginner● High impactGEO / AI
Illustration: PerplexityBot access fix guide
GEO / AI · FIX GUIDE PerplexityBot access IMPACTEFFORT TIME15 minutes LEVELBeginner

Why it matters

Perplexity builds its answers from pages that PerplexityBot has been allowed to crawl. Its cited sources appear as clickable links, so being crawlable is what makes your site eligible for that referral traffic. If the bot is blocked, Perplexity will usually describe your topic using competitors' pages or third-party summaries of your content.

Your brand loses the citation, the click and control over how your product is described. txt group plus one firewall exception, and it doesn't affect Googlebot or other crawlers.

✕Beforehttps://example.com/robots.txtcontains "User-agent: PerplexityBot"followed by "Disallow: /" (or✓Afterrobots.txt has a "User-agent:PerplexityBot" group with "Allow: /",and the server returns 200 to requests

What you need

Edit access to your robots.txt (CMS SEO settings, theme files, or web root)
Access to your CDN/WAF dashboard if you use one (e.g. Cloudflare, Sucuri, AWS WAF)
A terminal with curl, or an online robots.txt tester
Your sitemap URL (e.g. https://example.com/sitemap.xml)

Step-by-step fix

1

Fetch your live robots.txt

Start with the file crawlers actually receive, not the one in your CMS. A cache, plugin or CDN can serve something different from what you think you saved. txt in a browser, or fetch it from the command line.

Confirm it returns HTTP 200 with Content-Type: text/plain. A 5xx error or an HTML error page can cause crawlers to treat the whole site as off-limits.

text
curl -sI https://example.com/robots.txt
curl -s https://example.com/robots.txt
Pro tip: Check both the www and non-www hosts, and http as well as https. Each host serves its own robots.txt.
2

Find every rule that blocks PerplexityBot

Robots.txt groups apply to the user agents listed directly above them. A crawler obeys the most specific group that matches its name and ignores the * group. Look for:

  • A User-agent: PerplexityBot group with Disallow: /
  • PerplexityBot stacked with other AI bots above a single Disallow: /
  • No PerplexityBot group at all, combined with User-agent: * and Disallow: /
  • Plugin-generated "block AI crawlers" blocks, often labelled with a comment
text
# These patterns all block PerplexityBot
User-agent: PerplexityBot
Disallow: /

User-agent: GPTBot
User-agent: PerplexityBot
User-agent: ClaudeBot
Disallow: /

User-agent: *
Disallow: /
Pro tip: Search the file for "perplexity" in any case. Some setups use `Perplexitybot` or wildcard tricks that are easy to miss.
3

Add an explicit PerplexityBot allow group

Remove PerplexityBot from any blocking group. Then add its own group with Allow: /. If you need to keep private paths out, such as /cart/ or /account/, add those Disallow lines inside the PerplexityBot group too.

Because the bot ignores the * group once it has its own, it won't inherit them from there. Keep the Sitemap line so the bot can find your URLs.

text
# robots.txt for example.com
User-agent: PerplexityBot
Allow: /
Disallow: /cart/
Disallow: /account/

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: *
Allow: /
Disallow: /cart/
Disallow: /account/

Sitemap: https://example.com/sitemap.xml
Pro tip: Put a blank line between groups and don't leave a line like `Disallow:` with nothing after it inside a group you meant to block. An empty value means "allow everything", which confuses reviewers.
4

Publish the change where robots.txt is generated

Save the change in the system that actually produces the file:

  • A physical robots.txt in your web root
  • Your SEO plugin's file editor
  • A theme template
  • Framework code

If both a physical file and a plugin exist, the physical file usually wins on WordPress, so edit that one. Then purge the robots.txt URL from your page cache and your CDN cache.

Pro tip: See the platform notes below for the exact editor location on WordPress, Shopify, Wix, Webflow and Next.js.
5

Allow PerplexityBot through your CDN or WAF

Robots.txt only grants permission. A firewall can still return 403 or a JavaScript challenge, which the crawler cannot solve. Check these places:

  • AI-crawler blocking toggles
  • Bot-fight modes
  • Rate limits
  • Custom rules that match "bot" or "crawler" in the user agent

Create a skip/allow rule for requests whose user agent contains PerplexityBot. Where your WAF supports it, pair that with Perplexity's published IP ranges (https://www.perplexity.com/perplexitybot.json) so spoofed user agents aren't let through.

nginx
# Cloudflare custom rule (Security → WAF → Custom rules), action: Skip
(http.user_agent contains "PerplexityBot" and ip.src in $perplexity_ips)

# Nginx: make sure no UA block catches it
if ($http_user_agent ~* "PerplexityBot") { set $allow_ai 1; }
Pro tip: Create `$perplexity_ips` as an IP list in Cloudflare (Manage Account → Configurations → Lists), filled from the perplexitybot.json file. Re-check that file periodically, because the ranges change.
6

Test the response as PerplexityBot

Request both robots.txt and a normal content page using PerplexityBot's user agent string. Both should return 200 with real HTML content, not a 403, 429, a challenge page or an empty body. A pass here means at least your user-agent rules are fine.

text
UA="Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)"
curl -s -o /dev/null -w "%{http_code}\n" -A "$UA" https://example.com/robots.txt
curl -s -o /dev/null -w "%{http_code}\n" -A "$UA" https://example.com/blog/sample-post/
Pro tip: Your own IP isn't in Perplexity's ranges. If your WAF verifies IPs, a 200 here doesn't prove the real bot gets through, so also check your WAF event log.
7

Re-run the audit and watch your logs

Run the audit again to confirm PerplexityBot is now reported as allowed. Over the next days and weeks, filter your server or CDN logs for PerplexityBot to confirm real crawl hits are returning 200. How soon and how often it crawls varies by site.

text
grep -i "PerplexityBot" /var/log/nginx/access.log | awk '{print $9}' | sort | uniq -c

Platform-specific notes

WordPress

In Yoast SEO, go to Yoast SEO → Tools → File editor. In Rank Math, go to Rank Math → General Settings → Edit robots.txt. If a physical robots.txt exists in the web root, it overrides the plugin's virtual file, so edit it via SFTP instead. Some security plugins and "block AI bots" plugins add their own Disallow rules or server blocks, so check their settings too.

Shopify

Go to Online Store → Themes → … → Edit code and add a `robots.txt.liquid` template under Templates. Keep the default `{% for group in robots.default_groups %}` loop so Shopify's built-in rules stay. Then add a plain-text `User-agent: PerplexityBot` / `Allow: /` group after the loop.

Wix

Go to Marketing & SEO → SEO → Robots.txt Editor (under Tools and settings). Add the PerplexityBot group and save. Wix serves the file directly, and there's no separate firewall for you to configure.

Next.js

With the App Router, create `app/robots.ts` that exports a function returning `MetadataRoute.Robots`. For example: `rules: [{ userAgent: 'PerplexityBot', allow: '/' }, { userAgent: '*', allow: '/' }]` plus `sitemap: 'https://example.com/sitemap.xml'`. Delete any static `public/robots.txt`, because it conflicts with the generated file. Also check middleware and hosting firewalls, such as Vercel Firewall rules, for user-agent blocks.

Cloudflare

Check Security → Bots and the AI Crawl Control section (formerly AI Audit). Cloudflare can block AI crawlers by default on newer zones, and it can also inject Disallow rules into a managed robots.txt. Make sure PerplexityBot is set to Allow there. Then add a WAF custom rule with the Skip action for its user agent combined with Perplexity's IP list.

Check that it worked

✅ How to verify

  • Open https://example.com/robots.txt and confirm a `User-agent: PerplexityBot` group with `Allow: /` is present and no group that applies to it says `Disallow: /`.
  • Run curl with the PerplexityBot user agent against robots.txt and a content page; both should return HTTP 200 with real content.
  • Check your CDN/WAF event log for requests from PerplexityBot and confirm none are blocked or challenged.
  • Re-run the audit; the PerplexityBot access check should pass.

⚠️ Common mistakes

  • Adding `Allow: /` under `User-agent: *` while a separate PerplexityBot group still says `Disallow: /`. The named group wins.
  • Fixing robots.txt but leaving a Cloudflare AI-bot block, Bot Fight Mode or a WAF rule that returns 403 or a challenge to the crawler.
  • Editing a plugin's virtual robots.txt while a physical robots.txt file in the web root overrides it, or forgetting to purge the cached file.
  • Allowing any request whose user agent contains "PerplexityBot" without IP verification, which lets scrapers spoof it to bypass your protections.

Is this still an issue on your site? Run a free audit — ZORO checks this and 294 other SEO, AI-visibility, accessibility and security signals, with fixes for your platform.

Check my site free

FAQ

What is the difference between PerplexityBot and Perplexity-User?

PerplexityBot is the crawler that indexes pages so Perplexity can surface and cite them in answers, and it respects robots.txt. Perplexity-User fetches a page when a person asks Perplexity about a specific URL. Perplexity says it generally doesn't apply robots.txt to these user-initiated fetches. This audit check is about PerplexityBot.

Does allowing PerplexityBot let Perplexity train AI models on my content?

Perplexity describes PerplexityBot as a crawler that surfaces and links websites in its search results, not as a crawler for training foundation models. Review Perplexity's current documentation and terms if training use matters to you. Robots.txt controls crawling only.

How long until my site shows up in Perplexity answers after unblocking?

Crawling can start within days once the bot can reach you, but there's no fixed schedule. Appearing as a cited source also depends on whether your pages answer the question well and are easy to parse. A sitemap and fast, server-rendered HTML help.

Can I allow PerplexityBot on some sections but not others?

Yes. Inside the `User-agent: PerplexityBot` group, use `Allow: /` plus `Disallow:` lines for the paths you want excluded, such as `/members/` or `/checkout/`. Remember it won't inherit Disallow rules from the `*` group once it has its own group.