How to Fix a CDN or WAF Blocking AI Crawlers
Your robots.txt says AI crawlers such as GPTBot, ClaudeBot or PerplexityBot are welcome, but your CDN, WAF, hosting firewall or server config returns a 403, 429, 503 or drops the connection when they request a page, while Googlebot gets a normal 200. This guide shows how to reproduce the block, find the layer causing it, change the rule, and confirm every AI user agent gets the same response as a normal visitor.

Why it matters
AI assistants and answer engines can only cite, summarize or link to pages their crawlers can fetch. When a firewall blocks them, your content is left out of ChatGPT search, Claude, Perplexity and similar answers, and competitors get cited instead. txt allows the crawlers, so it often goes unnoticed.
The block usually comes from a one-click bot setting, a managed ruleset or a rate limit that someone enabled for scrapers. txt says yes and the server says no, which makes your actual AI policy hard to audit.
What you need
Step-by-step fix
Reproduce the block with each AI user agent
txt allows. Note the status code for each one. 403 means forbidden or challenged, 429 means rate limited, 503 usually means a challenge page or overload, and a timeout or reset means the connection was dropped.
Any AI agent that gets something other than the Googlebot result is the problem to fix.
for ua in "Googlebot/2.1 (+http://www.google.com/bot.html)" "GPTBot/1.2 (+https://openai.com/gptbot)" "OAI-SearchBot/1.0 (+https://openai.com/searchbot)" "ChatGPT-User/1.0 (+https://openai.com/bot)" "ClaudeBot/1.0 (+claudebot@anthropic.com)" "Claude-User/1.0" "PerplexityBot/1.0 (+https://perplexity.ai/perplexitybot)" "Perplexity-User/1.0" "CCBot/2.0 (https://commoncrawl.org/faq/)" "Amazonbot/0.1" "meta-externalagent/1.1"; do
code=$(curl -s -o /dev/null -w "%{http_code}" --max-time 15 -A "Mozilla/5.0 (compatible; $ua)" https://example.com/)
echo "$code $ua"
doneIdentify which layer is returning the error
Run curl -sI with a failing user agent and read the response headers. They show which system answered:
server: cloudflarewithcf-ray, orcf-mitigated: challenge, means Cloudflare.x-amzn-waf-*orx-amz-cf-idmeans AWS WAF/CloudFront.x-vercel-mitigatedmeans the Vercel Firewall.x-sucuri-idorx-sucuri-blockmeans Sucuri.server: AkamaiGHostmeans Akamai.
If you see only your origin's headers, such as server: nginx or server: Apache, the block is on your server or in a plugin. In that case, skip to step 5.
curl -sI -A "Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot)" https://example.com/ | grep -iE "^(HTTP|server|cf-|x-amzn|x-vercel|x-sucuri|retry-after)"
Turn off blanket AI-bot blocking on the CDN
Most CDNs now offer a one-click AI blocker, and some enable it by default for new zones. On Cloudflare, open the domain and go to AI Crawl Control → Crawlers. Set each crawler you want to Allow instead of Block.
Then go to Security → Settings and turn off Block AI bots if it is on. Check Bot Fight Mode or Super Bot Fight Mode as well, because it can challenge automated agents. Set Verified bots to Allow and Definitely automated to something other than Block.
On AWS WAF, edit the AWSManagedRulesBotControlRuleSet rule group and set the AI-related category rules (for example CategoryAI) to Count instead of Block.
Add an explicit allow/skip rule for AI crawlers
Custom WAF rules, geo-blocks and "block empty or unknown user agent" rules often catch AI crawlers by accident. Add a rule above them that skips the remaining custom rules, managed rules, Super Bot Fight Mode and rate limiting for verified AI crawlers. On Cloudflare, go to Security → Security rules → Create rule → Custom rule, choose action Skip, and use an expression based on the verified-bot category.
That way, attackers who only fake the user-agent string are not let through.
(cf.verified_bot_category in {"AI Crawler" "AI Search" "AI Assistant"}) or (cf.client.bot and http.user_agent contains "bot")Remove user-agent blocks on the origin server
htaccess` for AI bot names. Hosting "bad bot" lists and security plugins often contain GPTBot, ClaudeBot, CCBot or Bytespider. In WordPress, also check Wordfence → Firewall → Blocking, All In One WP Security → Firewall → User Agent Blocking, and any "block AI scrapers" plugin.
Delete or comment out the entries for crawlers you allow, then reload the server.
grep -RniE "gptbot|oai-searchbot|chatgpt-user|claudebot|claude-user|perplexity|ccbot|amazonbot|meta-externalagent|bytespider" /etc/nginx /etc/apache2 /var/www/example.com/.htaccess 2>/dev/null
# Typical rule to remove (Apache):
# RewriteCond %{HTTP_USER_AGENT} (GPTBot|ClaudeBot|PerplexityBot) [NC]
# RewriteRule .* - [F,L]
# Typical rule to remove (Nginx):
# if ($http_user_agent ~* (GPTBot|ClaudeBot|PerplexityBot)) { return 403; }
sudo nginx -t && sudo systemctl reload nginxRelax rate limits and timeouts that hit crawlers
A 429, a 503 or a dropped connection with no response usually comes from a rate-limit rule, a connection limit or a slow origin, not a deliberate block. Raise the threshold for verified bots, or exclude them, in your CDN rate-limiting rules (for example in Cloudflare's rate limiting rules, or the rate-based rule in AWS WAF). On Nginx, check the limit_req and limit_conn zones.
If your homepage takes more than a few seconds to generate, enable full-page caching so crawler requests are served from the edge.
# Nginx: give a higher burst to known crawlers instead of a hard limit
map $http_user_agent $limit_key {
default $binary_remote_addr;
"~*(GPTBot|OAI-SearchBot|ClaudeBot|PerplexityBot|Amazonbot)" "";
}
limit_req_zone $limit_key zone=perip:10m rate=10r/s;Make robots.txt match your real policy
If you do want to keep certain AI crawlers out, say so in robots.txt rather than blocking them silently at the firewall. Crawlers will respect the rule, your policy stays visible and auditable, and the audit will no longer count those bots as blocked by mistake. Allow only the crawlers you actually let through the firewall.
User-agent: GPTBot Allow: / User-agent: OAI-SearchBot Allow: / User-agent: ClaudeBot Allow: / User-agent: PerplexityBot Allow: / # Deliberately excluded User-agent: Bytespider Disallow: / Sitemap: https://example.com/sitemap.xml
Re-test every AI user agent
Purge the CDN cache, wait a minute for the rule changes to propagate, and rerun the loop from step 1. Every AI user agent allowed in robots.txt should return the same status as the Googlebot request, normally 200. Then check your CDN's security event log for the next few days and confirm that real crawler traffic shows as allowed, not challenged.
curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0 (compatible; ClaudeBot/1.0; +claudebot@anthropic.com)" https://example.com/Platform-specific notes
Cloudflare
AI blocking can be switched on in several places: the AI Crawl Control crawler list, the "Block AI bots" setting, Bot Fight Mode/Super Bot Fight Mode, custom security rules and rate limiting rules. Check all of them. Free plans have only Bot Fight Mode, which cannot be bypassed with a skip rule. If you need finer control, turn it off and rely on custom rules. If Pay per crawl is enabled, crawlers receive 402 responses by design.
WordPress
Blocks usually come from security plugins (Wordfence, All In One WP Security, Solid Security), "block AI" plugins, or host-level bot lists on managed hosts such as WP Engine, Kinsta or SiteGround. Check plugin firewall settings and the host's dashboard. On some hosts the bot list can only be changed through support.
Vercel / Next.js
In the Vercel dashboard, go to Project → Firewall and review the Bot Protection and AI Bots managed rules, plus any custom rules. Also check middleware.ts for user-agent checks that return NextResponse with a 403 status.
AWS CloudFront + WAF
The Bot Control managed rule group (AWSManagedRulesBotControlRuleSet) labels and can block AI bots by category. Override the relevant category rules to Count, or add a higher-priority rule that allows the verified-bot label for the crawlers you want.
Shopify / Wix / Squarespace
These platforms manage the CDN and firewall themselves, so there is no WAF panel to edit. If AI user agents get 403 or 429, contact platform support with the curl output. Check any third-party bot-protection app, and remove it if it is the cause.
Check that it worked
✅ How to verify
- Rerun the curl loop and confirm every AI user agent allowed in robots.txt returns the same status code as the Googlebot request (normally 200).
- Compare response sizes with `curl -s -A "GPTBot/1.2" https://example.com/ | wc -c` against a browser user agent, to make sure the crawler gets real HTML and not a challenge page served with a 200.
- Check your CDN or WAF event log over the following days for real GPTBot, ClaudeBot and PerplexityBot requests, and confirm they show as allowed, not blocked or challenged.
- Run the audit again and confirm this check passes.
⚠️ Common mistakes
- Fixing only the Cloudflare "Block AI bots" toggle while a custom security rule, a rate limit or a WordPress security plugin is still returning 403.
- Allowing any request whose user-agent string contains "GPTBot". Scrapers can fake that string, so allow verified-bot categories or published crawler IP ranges instead.
- Treating a 200 response as success when the body is actually a JavaScript challenge or CAPTCHA page.
- Forgetting to purge the CDN cache, so an old cached 403 or an old robots.txt is still served after the rule change.
Is this still an issue on your site? Run a free audit — ZORO checks this and 294 other SEO, AI-visibility, accessibility and security signals, with fixes for your platform.
Check my site freeFAQ
Why does my site block GPTBot when robots.txt allows it?
robots.txt is only a request that crawlers read; it doesn't control your firewall. CDNs and security tools have their own bot rules, and some of them block AI crawlers by default or through a one-click setting. The firewall rejects the request before robots.txt matters, so you have to change both to match.
Should I block AI crawlers or allow them?
It depends on your goals. If you want to be cited in ChatGPT search, Perplexity or Claude answers, allow the search and user-triggered agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) at a minimum. If you object to model training, you can disallow training crawlers such as GPTBot or CCBot in robots.txt. Either way, state the policy in robots.txt instead of blocking silently at the CDN.
Is it safe to let AI crawlers through my WAF?
Yes, if you allow them by verified identity rather than by user-agent string alone. Cloudflare's verified bot categories and AWS's verified bot labels check that requests come from the operator's published IP ranges, so impersonators are still filtered out.
Why do AI bots get a 429 error on my site?
A 429 means a rate limit was triggered. Crawlers can request many pages in a short burst and hit per-IP limits meant for abusive traffic. Raise the threshold or exempt verified crawlers in your rate-limiting rules, and serve cached pages so their requests put little load on your origin.
How to Create and Publish an llms.txt File for AI Tools
How to Set an Explicit Google-Extended Policy in robots.txt
How to Fix Content That Isn't Readable Without JavaScript
How to Allow PerplexityBot in robots.txt and Your Firewall
How to Get Your Brand Mentioned in ChatGPT Answers
How to Allow ClaudeBot and Claude-SearchBot in robots.txt